Guide
This project provides command-line tools to configure the JIRA/XRAY Test Execution and manage the upload of BDD test results in the form of cucumber JSON files (generated through pytest-bdd).
There are two tools:
xray-generate-configuration helps you to configure your package, so that the right BDD test results are uploaded to the right place with the right metadata. It generates a test execution configuration file, i.e. tests/xray-config.json. You will probably only ever run this tool once, to bootstrap an initial configuration file. After that, modifications to the configuration can be effected by manually editing the configuration file.
xray-upload performs the upload of BDD test results to Jira Xray.
Installing
How you install this package depends on how you intend to use it. Choose from one of these two use cases:
Developers will routinely upload test results as part of their development workflow. They need ongoing access to the xray-upload tool. In this case, you will want this package to be one of your package’s development dependencies, so that the xray-upload is available as part of your package’s development installation. If using poetry, for example, you would
me@local:~$ poetry add --group dev ska-ser-xray
Uploading of test results will be performed automatically in the CI/CD pipeline, rather than manually by developers. Developers do not need access to the xray-upload tool. In that case, you only need to install this package temporarily, in order to run the one-off xray-generate-configuration tool, and then you can uninstall it again:
me@local:~$ pip install ska-ser-xray me@local:~$ # use the xray-generate-configuration tool to bootstrap your configuration me@local:~$ pip uninstall ska-ser-xray
In either case, the test results to be uploaded will be generated by two pytest plugins:
pytest-bdd allows you to run BDD tests using pytest, and produces BDD test results in cucumber.json format
pytest-json-report produces test results in report.json format.
These two plugin packages must be installed in the environment in which you run your tests, e.g. your package’s development environment.
Configuration
When using the functionality for the first time it is necessary to create a configuration file which provides metadata for the XRAY Test Execution. It has defaults and can be customised through environment variables.
In the example below it can be interpreted as:
for the field name (line 1)
read the environment variable CI_COMMIT_BRANCH (line 2)
if it exists - use its value
if it doesn’t - use default as the value
look up the corresponding key in the definitions (line 4-5)
So if it isn’t found in the environment variables, the XRAY configuration will be:
name="branch execution"
1 "name": {
2 "env": "CI_COMMIT_BRANCH",
3 "maps_to": {
4 "default": "branch execution",
5 "main": "main branch execution"
6 }
A configuration file can be generated through xray-generate-configuration. There are three configuration items which should be provided to the command:
–project the XRAY enabled JIRA project to use
–chart-file the relative path to the Chart.yaml of the repository’s helm chart used for the BDD tests.
–test-plan-default a XRAY test plan issue id to link to by default. This is required to be of JIRA issue type Test Plan!
Example:
xray-generate-configuration -c charts/ska-example/Chart.yaml -p XTP -t "XTP-0000"
which creates tests/xray-config.json. This file needs to be committed. Values where the defaults weren’t provided on the command-line or didn’t resolve are set to %DEFINE_ME%. These will need to be replaced manually before a successful upload can happen.
Advanced configuration
For a custom project other than XTP there needs to be a little more configuration. In the config file (xray-config.json), custom projects as well as the settings for fields can be addressed. The example below modifies:
The hostname jira_url
The XRAY enabled project project.id
For the customfield_mapping
start_date and end_date don’t exist in the project
test_environments and test_plans have specific customfield_<n> mappings
1"jira_url": "https://jira-test.skatelescope.org",
2"project": {
3 "id": "TE",
4 "issuetype_id": 11602,
5 "customfield_mapping": {
6 "start_date": "",
7 "end_date": "",
8 "test_plans": "customfield_11927",
9 "test_environments": "customfield_11925"
10 }
11}
Note
For all https://jira.skatelescope.org hosted projects only the project.id needs to be changed from the generated defaults.
Usage
Once a configuration has been settled on, the upload functionality can be used in the project’s CI/CD pipeline, or manually.
As mentioned above, your project’s dependencies need to include:
and the pytest command needs to have the following extra arguments:
--cucumberjson=build/reports/cucumber.json \
--json-report --json-report-file=build/reports/report.json
Note
The file locations build/reports/cucumber.json and build/reports/report.json are defaults for the tools in this project. If you choose different ones (not fully supported yet), you will need to modify your configuration file, your Makefile, and your CI/CD pipeline configuration.
gitlab-ci pipeline
In the project’s Makefile add:
include .make/xray.mk
In the project’s .gitlab-ci.yml add:
include:
- project: 'ska-telescope/templates-repository'
file: 'gitlab-ci/includes/xray-publish.gitlab-ci.yml'
(If the include: key is already present, append this new include to the existing list of includes.)
This will add a manual step to the gitlab publish stage.
Note
Currently only the default locations for configuration and test artefacts are used. This can’t be controlled through Makefile` variables yet.
Note
There is no need to provide Jira credentials in your CI pipeline configuration; these are already set in the pipeline environment.
Manual uploads
Once the test has been executed the results can also be uploaded manually using JIRA credentials.
Use –dry-run and –verbose to check for any errors first:
xray-upload -f tests/data/cucumber.json --dry-run --verbose
and then:
xray-upload -f tests/data/cucumber.json --verbose --username 'me@ad.skatelescope.org'