Guide

This project provides command-line tools to configure the JIRA/XRAY Test Execution and manage the upload of BDD test results in the form of cucumber JSON files (generated through pytest-bdd).

To achieve this, three files need to be present.

execution configuration file, i.e. tests/xray-config.json. It can be bootstrapped using the xray-generate-configuration tool provided though this package.
the BDD test results in form of build/reports/cucumber.json. It is generated by pytest-bdd and its naming and location controlled through the CICD Makefiles.
a pytest results file build/reports/report.json. It is generated by the pytest-json-report plugin and its naming and location controlled through the CICD Makefiles.

Configuration

When using the functionality for the first time it is necessary to create a configuration file which provides metadata for the XRAY Test Execution. It has defaults and can be customised through environment.

In the example below it can be interpreted as:

for the field name (line 9)
read the environment variable CI_COMMIT_BRANCH (line 10)
if it exists - use its value
if it doesn’t - use default as the value
look up the corresponding key in the definitions (line 12-13)

So if isn’t found in the environment variables XRAY configuration will be:

name="main branch execution"

example-config.json section

  "name": {
    "env": "CI_COMMIT_BRANCH",
    "maps_to": {
      "default": "branch execution",
      "main": "main branch execution"
    }

A configuration file can be generated through xray-generate-configuration. There are two configuration items which _should_ be provided to the command:

–chart-file the relative path to the Chart.yaml of the repository’s helm chart used for the BDD tests.
–test-plan-default a XRAY test plan issue id to link to by default.

Example:

xray-generate-description -o -c charts/ska-example/Chart.yaml -t "XTP-0000"

which creates tests/xray-config.json. This file needs to be committed.

Usage (in pipelines)

Once a configuration has been settled on, the upload functionality can be used in the project’s CICD pipeline (or manually).

Your project’s dependencies for running tests need to include:

pytest-bdd
pytest-json-report

and the pytest command needs to have the following extra arguments:

--cucumberjson=build/reports/cucumber.json \
--json-report --json-report-file=build/reports/report.json

Note that these two file locations are defaults for the tools in this project. You will need to modify your configuration file and the pipeline usage if you chose different ones.

gitlab-ci pipeline

Note

yet to be implemented

In the project’s Makefile add:

include .make/xray.mk

In the projects .gitlab-ci.yml add:

- project: 'ska-telescope/templates-repository'
  file: 'gitlab-ci/includes/xray-upload.gitlab-ci.yml'

Manual uploads

Once the test has been executed the results can also be uploaded manually using JIRA credentials.

Use –dry-run and –verbose to check for any errors first:

xray-upload -f tests/data/cucumber.json --dry-run --verbose

and then:

xray-upload -f tests/data/cucumber.json --verbose --username 'me@ad.skatelescope.org'

Tools

xray-upload

Uploads a cucumber results file to JIRA by way of the XRAY extension. Either use username/password or set the environment variable JIRA_AUTH for token authentication.

usage: xray-upload [-h] -f CUCUMBER_FILE [-u USERNAME] [-p PASSWORD]
                   [-e EXECUTION_CONFIGURATION] [-d] [-v]

options

-f, --cucumber-file

Path to the cucumber JSON results file.

-u, --username

JIRA account username

Default: “”

-p, --password

Password for the JIRA user. If not specified and the environment variable JIRA_AUTH is not set then you will be prompted for one.

Default: “”

-e, --execution-configuration

Path to test execution configuration json file (as defined by user)

Default: tests/xray-config.json

-d, --dry-run

Do not upload, just print artefacts

Default: False

-v, --verbose

Verbose output

Default: False

xray-generate-configuration

Generates a (sensible) default executionconfiguration file which describes the metadatato pass to JIRA/XRAY.

usage: xray-generate-configuration [-h] [-o OUTPUT_FILE] [-d] -c CHART_FILE
                                   [-t TEST_PLAN_DEFAULT]

options

-o, --output-file

File name of output file

Default: tests/xray-execution.json

-d, --dry-run

Don’t write file just print output

Default: False

-c, --chart-file

Name of the chart (file) against which the tests are executed.e.g. charts/ska-example/Chart.yaml

-t, --test-plan-default

The default JIRA test plan id, e.g. ‘XTP-1234’