How BDD testing works#

Understand BDD concepts, the SKA testing architecture, and how tests connect to requirements.

What is BDD?#

BDD (Behaviour Driven Development) describes system behaviour using plain language. Tests focus on what the system does in response to specific inputs, not how it works internally.

BDD testing typically covers component interactions and system-level behaviour. For where it sits among the other levels of testing, see Software Testing Policy and Strategy.

BDD tests use the Gherkin specification language with three core keywords:

Given — the initial context or state
When — an action or event
Then — the expected outcome

This example tests a login page:

Given the SKA Community Confluence website
And I am not logged in
When I click on the login button
Then I see a login page

Developers, testers, project managers, and stakeholders can all read and understand this test. This shared understanding is BDD’s key benefit.

Why SKA uses BDD testing#

BDD testing addresses specific SKA needs:

Formal requirements verification: The SKA has thousands of formal requirements at L1 (observatory), L2 (sub-system), and interface levels. BDD tests provide clear traceability from implementation back to requirements.
Stakeholder communication: Plain language test definitions enable Feature Owners, Product Owners, and verification engineers to review and contribute to test design without reading code.
Living documentation: BDD tests serve as executable specifications. When tests pass, they document current system capabilities. When they fail, they identify gaps between requirements and implementation.
Integration testing: The SKA integrates software with extensive hardware across multiple sub-systems. BDD tests describe expected interactions at component and system boundaries.
Automated verification: BDD tests run automatically in CI/CD pipelines, continuously verifying requirements and catching regressions early.

The SKA BDD architecture#

This diagram shows how BDD tests flow from requirements to automated verification:

BDD test workflow diagram — BDD testing workflow: from requirements through JIRA to automated CI/CD execution.#

The workflow:

ICDs and JAMA requirements import into JIRA (L1, L2, IFID, VTS projects)
Teams define Gherkin tests in JIRA’s XTP project and link them to requirements
Teams export feature files from JIRA and add them to GitLab repositories
pytest-bdd implements the test steps in Python
CI/CD pipelines run tests and output results as JSON
XRay (JIRA plugin) imports results and updates requirement statuses

Steps 2 and 3 can also run in the opposite direction: teams write the feature files in the repository first, then register them in JIRA with the ska-ser-xray tooling, which creates the Test issues and writes their keys back into the files — see BDD how-to guides.

This creates a complete traceability chain from formal requirements to automated test results.

For the make targets and pipeline templates that implement steps 5 and 6, see BDD reference.

JIRA organisation#

Understanding BDD test organisation in JIRA helps you navigate the system effectively.

Requirements hierarchy#

Requirements live in dedicated JIRA projects:

L1 — Observatory-level requirements
L2 — Sub-system requirements
IFID — Interface requirements between sub-systems
VTS — Verification requirements

JAMA remains the source of truth for formal requirements.

Test organisation#

The XTP project organises tests:

Test: An individual test case with Gherkin steps. Each test verifies one specific behaviour.
Test Set: A collection of tests that together verify a requirement, or otherwise provide a logical grouping for tests. One requirement typically has one Test Set, but can have multiple Test Sets for different aspects.
Test Plan: Groups tests (which may or may not be in Test Sets) for planning and reporting across executions. The ska-ser-xray tooling attaches tests and their executions to a Test Plan.
Test Execution: Records the results of running tests. CI/CD pipelines create these automatically.

Example structure#

Consider verification requirement VTS-221, which demonstrates a sub-array performing an imaging scan:

VTS-221 (Requirement)
  └── XTP-494 (Test Set) — "tests" relationship
        ├── XTP-417 (Test) — Sub-array resource allocation
        ├── XTP-427 (Test) — Sub-array transitions from IDLE to READY state
        ├── XTP-428 (Test) — Sub-array performs an observational imaging scan
        └── XTP-436 (Test) — Sub-array deallocation of resources

The Test Set groups related tests. You can reuse individual tests across multiple Test Sets, and link Test Sets to multiple requirements.

How results flow back to JIRA#

When CI/CD pipelines run BDD tests:

pytest-bdd executes the tests and generates a JSON report
The pipeline uploads the JSON to JIRA via the XRay API
XRay creates Test Execution tickets showing pass/fail status
Test and Requirement tickets update to reflect current status
Dashboards aggregate results across requirements

This automation means requirement verification status stays current without manual updates.

Test results on a VTS ticket — A VTS ticket showing the status of associated tests.#

When to use BDD vs standard testing#

BDD testing adds value in specific situations:

Use BDD when:

Verifying formal SKA requirements
Tests need to be understood by non-developers
Traceability to JIRA is required
Testing component interactions or system-level behaviour
Documentation of system capabilities is valuable

Use standard pytest when:

Unit testing internal functions
Testing implementation details
Rapid iteration during development
JIRA traceability is not needed

You can use BDD style (given/when/then) with pytest-bdd without JIRA integration, giving you readable tests without generating JIRA tickets.

Tip

Use PI planning to identify requirements or features that benefit from BDD tests. Work with Feature Owners and verification colleagues to define effective test scenarios.

Test ownership#

BDD tests are a shared responsibility between Feature Owners, Product Owners, and the development teams.

Designing a good test needs domain expertise: understanding the acceptance criteria, the architecture, and the ways the system can fail. Feature and Product Owners are best placed for this — and for reading execution reports and judging whether a failure is a real problem. Implementing and maintaining the step definitions needs developers experienced with the test framework. The best results come from treating test design as a negotiation between the two: the domain expert formulates the behaviour, the developer chooses the API through which the test exercises the system.

Writing good BDD tests#

Effective BDD tests require domain expertise and understanding of what can go wrong.

Know the system under test: Understand the expected behaviour, edge cases, and failure modes. Cover the unhappy paths too — unresponsive devices, timeouts, recovery after failure — not only the sequence where everything works. Work with domain experts to identify what matters.
Use consistent phrasing: Reuse existing step phrasing from other tests. Check existing tests in the Xray step library before writing new steps. Consistency enables step reuse: where a step is reused in the same git repository, its implementation is reused too.
Keep steps atomic: Each step performs one action or checks one thing. Avoid compound steps. If you need to check more than one condition, use the “And” keyword. Using “And” for “When” steps is not recommended, as it means the test is testing more than one thing.
Keep steps declarative: Describe what happens, not how, and use domain language that stakeholders understand. When I configure the subarray is better than When I send a JSON payload to the REST endpoint.
Test one behaviour per scenario: Each scenario verifies one specific behaviour. Use multiple scenarios for different aspects of a requirement.
Start every scenario from a known state: Define an invariant condition that holds before each scenario — for example, the telescope in STANDBY with all subarrays OFF — and establish it with pytest fixtures, so scenarios run independently and in any order. Prefer fixtures over Gherkin Background for this. Also check that those conditions do in fact hold before continuing with the test.
Include enough context: The scenario title and steps together tell the complete story of what the test verifies. Readers must understand the test purpose without external documentation.
Collaborate on test design: Feature Owners, Product Owners, testers, and developers all contribute different perspectives. The best tests come from collaborative definition.

Compare a well-formed scenario with a poor one:

# Good - clear, reusable steps
Given the telescope is in STANDBY state
When I send the Configure command with valid parameters
Then the subarray enters READY state

# Poor - vague, compound, not reusable
Given everything is set up
When I do the thing and check the result
Then it works

Current limitations#

The BDD testing system has known limitations:

Two representations to keep in sync: Tests exist both as feature files in git and as issues in JIRA. The ska-ser-xray tooling automates the repository-to-JIRA direction, but which copy is the source of truth remains a per-team decision.
Test execution volume: Running tests on every branch generates thousands of JIRA tickets. Run JIRA integration only on main/master branches of integration repositories.
JAMA synchronisation: The process for integrating new verification requirements with JAMA is still evolving.
Dashboard complexity: As the number of tests grows, maintaining useful dashboards requires ongoing curation.

We are addressing these limitations as the system matures.