BDD how-to guides#

Task-oriented recipes for creating and implementing BDD tests. To create your first BDD test end to end, work through BDD tutorial first.

There are two ways to create tests: write feature files in your repository and register them in JIRA with the ska-ser-xray tooling, or write tests in JIRA and export them as feature files. Both paths end the same way — feature files in git, implemented with pytest-bdd.

Create tests from the repository#

Write feature files in git and register them in JIRA with the ska-ser-xray tooling. The tooling creates the Test and Test Plan issues for you and writes their keys back into the feature files as tags.

Install the tooling#

pip install ska-ser-xray

Set up JIRA access#

Generate a personal API token in your JIRA profile, then set the environment variables the commands read:

export JIRA_URL="https://jira.skatelescope.org/"
export JIRA_PROJECT_KEY="XTP"
export JIRA_USER="<your-jira-username>"
export JIRA_AUTH="<your-api-token>"

Warning

The API token is personal. Never commit it or add it to your project’s CI/CD variables — pipelines receive their own JIRA_AUTH from the shared CI configuration.

Create the Test Plan#

xray-testplan-setup "<test plan name>" --create-if-not-exists

The command prints the Test Plan key — for example XTP-<id>, though test plans are not limited to the XTP project. You need it in the next two steps.

Generate the execution configuration#

xray-generate-configuration -c charts/<your-chart>/Chart.yaml -t XTP-<id>

This writes tests/xray-config.json, which selects the Test Plan and maps pipeline variables to test environments, labels, and versions. Commit it to the repository.

Register the tests#

xray-test-setup tests/bdd/features/*.feature --test-plans XTP-<id> --override-input-files

The command creates a Test issue in XTP for each scenario, links it to the Test Plan, and writes the XRay tags back into your local feature files. Commit the updated files. Re-run it whenever you add or amend scenarios; use --dry-run first to preview the changes.

Create a BDD test in JIRA#

This workflow authors tests in JIRA and exports them as feature files, keeping full traceability to SKA requirements. Work through the four stages in order.

Identify the requirement#

Start with an SKA requirement or Feature:

  • Use an existing requirement (L1, L2, interface, or verification)

  • Or create a new verification requirement in the VTS project

If creating a new requirement, label it with the PI (Program Increment) in which you plan to implement it.

Create a Test Set#

  1. Create a JIRA issue of type Test Set in the XTP project

  2. Add a fix version for the relevant PI (optional)

  3. Link the Test Set to your requirement using the tests relationship

    • From the requirement side, use tested by instead

Create individual tests#

For each test in your Test Set:

  1. Create an issue of type Test in the XTP project

  2. Add fix version (optional)

  3. Click the Test Details tab

  4. Set:

    • Test type: Cucumber

    • Cucumber type: scenario

    • Cucumber scenario: Write your Gherkin steps (given, when, then)

  5. Link your test to the relevant Test Set

Tip

Reuse existing tests by linking them to new Test Sets. Check existing tests in XTP before creating duplicates.

Export the feature file#

After defining all tests for the Test Set:

  1. Open the Test Set in JIRA

  2. Click the More dropdown menu

  3. Select Export to Cucumber

  4. Save the .feature file

Repeat for each Test Set you need to exercise.

Implement tests with pytest-bdd#

Add the .feature files to your GitLab repository and implement the test steps. A typical layout keeps feature files next to the step modules that implement them:

tests/bdd/
├── conftest.py            # Shared fixtures and shared step definitions
├── features/
│   ├── monitor.feature
│   └── attributes.feature
├── test_monitor.py        # Steps for monitor.feature
└── test_attributes.py     # Steps for attributes.feature

Set up your test file#

Create a test module that imports pytest-bdd and loads your scenarios:

from pytest_bdd import given, when, then, scenarios, parsers
import pytest

# Load scenarios from the feature file
scenarios('path/to/your_test.feature')

# Create a fixture to pass data between steps
@pytest.fixture
def context():
    return {}

Write step definitions#

Annotate your test methods with Gherkin keywords:

@given('I have an SDPSubarray device')
def subarray_device(devices, context):
    """Get a subarray device for testing."""
    context['device'] = devices.get_device(DEVICE_NAME)

@when('I set the device state to ON')
def set_device_state(context):
    """Set the device to ON state."""
    context['device'].on()

@then('the device state is ON')
def check_device_state(context):
    """Verify the device is in ON state."""
    assert context['device'].state == 'ON'

Reuse steps across tests#

Share step definitions across multiple tests by placing common steps in a conftest.py file:

# conftest.py
from pytest_bdd import given

@given('a running SDPSubarray device', target_fixture='subarray_device')
def subarray_device(devices):
    """Shared step - available to all tests in this directory."""
    return devices.get_device(DEVICE_NAME)

target_fixture makes the returned device available to later steps as the subarray_device fixture — without it, pytest-bdd discards the return value. Define each step in one place only: a definition in a test module silently shadows a conftest.py definition with the same text.

Handle parameterised tests#

Use the parsers module for tests with variable inputs:

import pytest
from pytest_bdd import when, then, parsers

@when(parsers.parse('OET create is given a {file} that does not exist'))
def file_not_found(file, context):
    """Attempt the create and keep the raised error for the Then step."""
    with pytest.raises((FileNotFoundError, ValueError)) as excinfo:
        oet.create(file)
    context['exception'] = excinfo.value

@then(parsers.parse('the OET returns an {error}'))
def check_error(error, context):
    """Verify the correct error is returned."""
    assert error in str(context['exception'])

Feature files support Examples tables for multiple test cases:

@XTP-1158
Scenario Outline: OET handles invalid files
  Given the Observation Execution Tool create command
  When OET create is given a <file> that does not exist
  Then the OET returns an <error>

  Examples:
    | file                    | error                                              |
    | file:///FileNotFound.py | FileNotFoundError: No such file or directory       |
    | sdljfsdjkfhsd           | ValueError: Script URI type not handled            |

Run tests and upload results#

Run tests locally#

Run your BDD tests through the SKA pipeline machinery so local runs match CI:

make python-test PYTHON_TEST_FILE=tests/bdd

To produce the Cucumber JSON report that JIRA needs, set the pytest-bdd reporting flag once in your Makefile — python-test appends it to every run, locally and in CI:

PYTHON_VARS_AFTER_PYTEST = --cucumberjson=build/reports/cucumber.json

Upload results to JIRA#

Uploading is handled by the xray-publish target from the .make submodule, not by a hand-written pipeline job. Wire it into your project as follows:

  1. Add the XRay targets to your Makefile — base.mk does not pull them in:

    include .make/python.mk
include .make/xray.mk

  2. Set PYTHON_VARS_AFTER_PYTEST in your Makefile as shown above, so the CI test job writes build/reports/cucumber.json; copy reports in after_script so they survive a failing test run — see How to Work with CI/CD at SKAO

  3. Generate tests/xray-config.json with xray-generate-configuration and commit it — see Create tests from the repository

  4. Include the XRay CI template in .gitlab-ci.yml:

    include:
  - project: 'ska-telescope/templates-repository'
    file: 'gitlab-ci/includes/xray-publish.gitlab-ci.yml'

The pipeline then runs make xray-publish, which uploads the results to JIRA, where XRay creates Test Execution tickets and updates the Test and Requirement statuses.

See BDD reference for the variables the target accepts.

Warning

Only run the JIRA integration on the main/master branch of repositories doing significant integration work. This prevents flooding JIRA with thousands of test execution tickets.

For other repositories, view test results in the standard GitLab CI/CD pipeline output.

Was this page helpful?