BDD reference

Gherkin syntax, JIRA ticket types, code templates, ska-ser-xray tooling, and CI/CD configuration.

For guidance on writing effective Gherkin, see How BDD testing works.

Gherkin syntax

Gherkin defines test scenarios using natural language keywords.

Keywords

Keyword	Purpose	Example
Given	Set up initial state/context	Given the SKA Community Confluence website
When	Perform an action	When I click on the login button
Then	Verify expected outcome	Then I see a login page
And	Continue previous step type	And I am not logged in
But	Negative continuation	But I do not see an error message

Scenario structure

@XTP-1156
Scenario: Descriptive name of the test
  Given some initial context
  And additional context
  When an action occurs
  And another action
  Then expected outcome
  And additional verification

Scenario Outline (parameterised)

Use Scenario Outline with an Examples table for multiple test cases:

@XTP-1157
Scenario Outline: Handle invalid inputs
  Given the system is ready
  When I provide <input>
  Then I receive <output>

  Examples:
    | input    | output          |
    | valid    | success         |
    | invalid  | error message   |
    | empty    | validation fail |

Background

Background steps run before every scenario in a feature file:

Feature: Status propagation

  Background:
    Given a clean database

  Scenario: Project status trickles down
    Given a Project containing an SBDefinition
    When the Project status is updated to "Submitted"
    Then the SBDefinition status is "Submitted"

For establishing initial state, SKAO practice prefers pytest fixtures — see How BDD testing works.

JIRA ticket types

Ticket type	Project	Purpose
Requirement	L1, L2, IFID, VTS	Source requirements JAMA imports
Feature	Various	User-defined features for PI planning
Test Set	XTP	Groups related tests for a requirement
Test	XTP	Individual BDD test with Gherkin steps
Test Plan	XTP	Groups tests for planning and reporting; ska-ser-xray attaches tests and executions to it
Test Execution	XTP	Records test run results (CI/CD auto-generates these)

JIRA projects

Project	Name	URL
L1	Level 1 Requirements	jira.skatelescope.org/projects/L1
L2	Level 2 Requirements	jira.skatelescope.org/projects/L2
IFID	Interface Requirements	jira.skatelescope.org/projects/IFID
VTS	Verification Requirements	jira.skatelescope.org/projects/VTS
XTP	Tests	jira.skatelescope.org/projects/XTP
BXT	Tests	jira.skatelescope.org/projects/BXT
CLT	Tests	jira.skatelescope.org/projects/CLT
AT	Tests	jira.skatelescope.org/projects/AT
MTP	Tests	jira.skatelescope.org/projects/MTP

pytest-bdd code templates

Basic test structure

"""BDD tests for [feature name]."""

from pytest_bdd import given, when, then, scenarios, parsers
import pytest

# Load all scenarios from the feature file
scenarios('path/to/feature_file.feature')


@pytest.fixture
def context():
    """Fixture to share data between steps."""
    return {}


@given('initial state description')
def setup_initial_state(context):
    """Set up the test context."""
    context['key'] = 'value'


@when('action description')
def perform_action(context):
    """Execute the action under test."""
    context['result'] = do_something(context['key'])


@then('expected outcome description')
def verify_outcome(context):
    """Assert the expected result."""
    assert context['result'] == expected_value

Parameterised steps

from pytest_bdd import when, then, parsers

@when(parsers.parse('I provide {input_value}'))
def provide_input(input_value, context):
    """Handle variable input from Examples table."""
    context['input'] = input_value


@when(parsers.re(r'I wait (?P<seconds>\d+) seconds'))
def wait_seconds(seconds, context):
    """Use regex for more complex patterns."""
    import time
    time.sleep(int(seconds))

Shared steps in conftest.py

# tests/bdd/conftest.py
"""Shared BDD step definitions."""

from pytest_bdd import given
import pytest


@pytest.fixture
def devices():
    """Fixture providing access to test devices."""
    return DeviceManager()


@given('I have a configured subarray', target_fixture='subarray')
def configured_subarray(devices):
    """Shared step; the return value becomes the ``subarray`` fixture."""
    subarray = devices.get_subarray()
    subarray.configure()
    return subarray

Feature file format

Structure feature files as follows:

# XSB-41.feature
# Exported from JIRA Test Set XSB-41

@XSB-41
Feature: Configure subarray for imaging scan
  As a telescope operator
  I want to configure a subarray
  So that I can perform an imaging observation

  @XSB-38
  Scenario: Configure subarray with valid parameters
    Given the telescope is in STANDBY state
    And I have a valid configuration file
    When I send the Configure command
    Then the subarray enters READY state
    And the configuration is applied

  @XSB-39
  Scenario: Handle invalid configuration
    Given the telescope is in STANDBY state
    When I send an invalid configuration
    Then the subarray returns an error
    And the subarray remains in STANDBY state

ska-ser-xray tooling

The ska-ser-xray Python package provides the XRay integration commands, including the xray-upload command that make xray-publish calls. Install it in your development environment (with pip, uv, or poetry, as your project uses) to manage Test and Test Plan issues from feature files:

pip install ska-ser-xray

Commands

Command	Purpose
xray-testplan-setup	Find or create Test Plan issues by name (--create-if-not-exists)
xray-generate-configuration	Generate tests/xray-config.json from a Helm chart and a Test Plan (-c <Chart.yaml> -t XTP-<id>)
xray-test-setup	Create a Test issue per scenario in local feature files and link it to Test Plans (--test-plans, --override-input-files, --dry-run)
xray-upload	Upload a test report to XRay (-f <report> -e <config>)
generate-steps-docs	Generate a Markdown reference for the test suite from feature files and step modules (generate-steps-docs <input folder> <output folder>)
xray-enrich-test-execution	Add links to the HTML test report, steps docs, and commit to Test Execution issues (--html-report, --test-docs, --commit-sha)

generate-steps-docs writes an index.md plus one Markdown file per feature file and per step module. Publish the output alongside your pipeline’s HTML report and pass both URLs to xray-enrich-test-execution — typically from the xray-post-publish hook — so every Test Execution links back to the run that produced it.

Environment variables

The commands read JIRA access from the environment (each also accepts equivalent flags):

Variable	Value
JIRA_URL	https://jira.skatelescope.org/
JIRA_PROJECT_KEY	Project holding the Test issues — XTP
JIRA_USER	Your JIRA username
JIRA_AUTH	Personal API token locally; supplied by the shared CI configuration in pipelines

Execution configuration

tests/xray-config.json is generated by xray-generate-configuration, not written by hand. It selects the Test Plan and maps pipeline variables to execution names, test environments, labels, and versions. Commit the generated file to the repository.

CI/CD configuration

Result upload is handled by the xray-publish target in the .make submodule (ska-cicd-makefile). Do not hand-write an upload job.

Makefile

base.mk does not pull in the XRay targets, so include xray.mk explicitly:

include .make/base.mk
include .make/python.mk
include .make/xray.mk

PYTHON_VARS_AFTER_PYTEST = --cucumberjson=build/reports/cucumber.json

python-test appends PYTHON_VARS_AFTER_PYTEST to every pytest run, so setting it here makes local runs and the CI test job both write the report to the path xray-publish reads (XRAY_TEST_RESULT_FILE).

Pipeline

Include the XRay template from templates-repository:

include:
  - project: 'ska-telescope/templates-repository'
    file: 'gitlab-ci/includes/xray-publish.gitlab-ci.yml'

Targets and hooks

Target	Purpose
xray-publish	Publish BDD test results from the repository to XRay
xray-pre-publish	Hook that runs before the upload — override in your Makefile
xray-post-publish	Hook that runs after the upload — override in your Makefile

The target runs xray-upload -f <result file> -e <execution config>.

Variables

Variable	Default	Purpose
XRAY_TEST_RESULT_FILE	build/reports/cucumber.json	Cucumber JSON report produced by the test run
XRAY_EXECUTION_CONFIG_FILE	tests/xray-config.json	Test execution configuration for the upload
XRAY_EXTRA_OPTS	empty	Additional switches passed to xray-upload
JIRA_AUTH	set by CI	JIRA/XRay API token — personal token locally, shared CI configuration in pipelines

Tip

When an upload misbehaves, run it with the verbose flag to see exactly what XRay received:

make xray-publish XRAY_EXTRA_OPTS="--verbose"

Generate the report

pytest-bdd writes the Cucumber JSON with --cucumberjson. Set the flag once in your Makefile, as shown above, rather than per command — a flag passed only on the command line never runs in CI, so xray-publish would find no report there. With the Makefile set:

make python-test PYTHON_TEST_FILE=tests/bdd

Note

In pipelines, JIRA_AUTH comes from the shared CI configuration — do not add JIRA credentials to your project’s CI/CD variables. For local runs of the ska-ser-xray commands, use a personal API token and never commit it. See Secret & Configuration Management.

Warning

Run the upload only on the main/master branch of repositories doing significant integration work. Running it on every branch generates thousands of Test Execution tickets.