Guide
This project provides command-line tools to configure the JIRA/XRAY Test Execution and manage the upload of BDD test results in the form of cucumber JSON files (generated through pytest-bdd).
To achieve this, three files need to be present:
test execution configuration file, i.e. tests/xray-config.json. It can be bootstrapped using the xray-generate-configuration tool provided though this package.
the BDD test results in form of build/reports/cucumber.json. It is generated by pytest-bdd and its naming and location controlled through the CICD Makefiles.
a pytest results file build/reports/report.json. It is generated by the pytest-json-report plugin and its naming and location controlled through the CICD Makefiles.
Configuration
When using the functionality for the first time it is necessary to create a configuration file which provides metadata for the XRAY Test Execution. It has defaults and can be customised through environment variables.
In the example below it can be interpreted as:
for the field name (line 1)
read the environment variable CI_COMMIT_BRANCH (line 2)
if it exists - use its value
if it doesn’t - use default as the value
look up the corresponding key in the definitions (line 4-5)
So if it isn’t found in the environment variables, the XRAY configuration will be:
name="branch execution"
1 "name": {
2 "env": "CI_COMMIT_BRANCH",
3 "maps_to": {
4 "default": "branch execution",
5 "main": "main branch execution"
6 }
A configuration file can be generated through xray-generate-configuration. There are three configuration items which should be provided to the command:
–project the XRAY enabled JIRA project to use
–chart-file the relative path to the Chart.yaml of the repository’s helm chart used for the BDD tests.
–test-plan-default a XRAY test plan issue id to link to by default.
Example:
xray-generate-description -o -c charts/ska-example/Chart.yaml -p XTP -t "XTP-0000"
which creates tests/xray-config.json. This file needs to be committed. Values where the defaults weren’t provided on the command-line or didn’t resolve are set to %DEFINE_ME%. These will need to be replaced manually before a successful upload can happen.
Usage (in pipelines)
Once a configuration has been settled on, the upload functionality can be used in the project’s CICD pipeline (or manually).
Your project’s dependencies for running tests and generating the necessary artefacts need to include:
and the pytest command needs to have the following extra arguments:
--cucumberjson=build/reports/cucumber.json \
--json-report --json-report-file=build/reports/report.json
Note that these two file locations are defaults for the tools in this project. You will need to modify your configuration file and the pipeline usage if you chose different ones (not fully supported yet).
gitlab-ci pipeline
In the project’s Makefile add:
include .make/xray.mk
In the projects .gitlab-ci.yml add:
- project: 'ska-telescope/templates-repository'
file: 'gitlab-ci/includes/xray-upload.gitlab-ci.yml'
This will add a manual step to the gitlab publish stage. Note currently only the default locations for the configuration and test artefacts are used. This can’t be controlled through Makefile` variables yet.
Manual uploads
Once the test has been executed the results can also be uploaded manually using JIRA credentials.
Use –dry-run and –verbose to check for any errors first:
xray-upload -f tests/data/cucumber.json --dry-run --verbose
and then:
xray-upload -f tests/data/cucumber.json --verbose --username 'me@ad.skatelescope.org'
Tools
xray-upload
Uploads a cucumber results file to JIRA by way of the XRAY extension. Either use username/password or set the environment variable JIRA_AUTH for token authentication.
usage: xray-upload [-h] -f CUCUMBER_FILE [-u USERNAME] [-p PASSWORD]
[-e EXECUTION_CONFIGURATION] [-d] [-v]
options
- -f, --cucumber-file
Path to the cucumber JSON results file.
- -u, --username
JIRA account username
Default: “”
- -p, --password
Password for the JIRA user. If not specified and the environment variable JIRA_AUTH is not set then you will be prompted for one.
Default: “”
- -e, --execution-configuration
Path to test execution configuration json file (as defined by user)
Default: tests/xray-config.json
- -d, --dry-run
Do not upload, just print artefacts
Default: False
- -v, --verbose
Verbose output
Default: False
xray-generate-configuration
Generates a (sensible) default executionconfiguration file which describes the metadatato pass to JIRA/XRAY.
usage: xray-generate-configuration [-h] [-o OUTPUT_FILE] [-d] -c CHART_FILE
[-t TEST_PLAN_DEFAULT] [-p PROJECT]
options
- -o, --output-file
File name of output file
Default: tests/xray-execution.json
- -d, --dry-run
Don’t write file just print output
Default: False
- -c, --chart-file
Name of the chart (file) against which the tests are executed.e.g. charts/ska-example/Chart.yaml
- -t, --test-plan-default
The default JIRA test plan id, e.g. ‘XTP-1234’
- -p, --project
The JIRA XRAY enabled project to use
Default: “XTP”