Guide

This project provides command-line tools to configure the JIRA/XRAY Test Execution and manage the upload of BDD test results in the form of cucumber JSON files (generated through pytest-bdd).

To achieve this, three files need to be present:

test execution configuration file, i.e. tests/xray-config.json. It can be bootstrapped using the xray-generate-configuration tool provided though this package.
the BDD test results in form of build/reports/cucumber.json. It is generated by pytest-bdd and its naming and location controlled through the CICD Makefiles.
a pytest results file build/reports/report.json. It is generated by the pytest-json-report plugin and its naming and location controlled through the CICD Makefiles.

Configuration

When using the functionality for the first time it is necessary to create a configuration file which provides metadata for the XRAY Test Execution. It has defaults and can be customised through environment variables.

In the example below it can be interpreted as:

for the field name (line 1)
read the environment variable CI_COMMIT_BRANCH (line 2)
if it exists - use its value
if it doesn’t - use default as the value
look up the corresponding key in the definitions (line 4-5)

So if it isn’t found in the environment variables, the XRAY configuration will be:

name="branch execution"

example-config.json section

  "name": {
    "env": "CI_COMMIT_BRANCH",
    "maps_to": {
      "default": "branch execution",
      "main": "main branch execution"
    }

A configuration file can be generated through xray-generate-configuration. There are three configuration items which should be provided to the command:

–project the XRAY enabled JIRA project to use
–chart-file the relative path to the Chart.yaml of the repository’s helm chart used for the BDD tests.
–test-plan-default a XRAY test plan issue id to link to by default.

Example:

xray-generate-description -o -c charts/ska-example/Chart.yaml -p XTP -t "XTP-0000"

which creates tests/xray-config.json. This file needs to be committed. Values where the defaults weren’t provided on the command-line or didn’t resolve are set to %DEFINE_ME%. These will need to be replaced manually before a successful upload can happen.

Advanced configuration

For a custom project other than XTP there needs to be a little more configuration. In the config file (xray-config.json), custom projects as well as the settings for fields can be addressed. The example below modifies:

The hostname jira_url
The XRAY enabled project project.id
For the customfield_mapping
- start_date and end_date don’t exist in the project
- test_environments and test_plans have specific customfield_<n> mappings

"jira_url": "https://jira-test.skatelescope.org",
"project": {
   "id": "TE",
   "issuetype_id": 11602,
   "customfield_mapping": {
      "start_date": "",
      "end_date": "",
      "test_plans": "customfield_11927",
      "test_environments": "customfield_11925"
   }
}

Note

For all https://jira.skatelescope.org hosted projects only the project.id needs to be changed from the defaults.

Usage (in pipelines)

Once a configuration has been settled on, the upload functionality can be used in the project’s CICD pipeline (or manually).

Your project’s dependencies for running tests and generating the necessary artefacts need to include:

and the pytest command needs to have the following extra arguments:

--cucumberjson=build/reports/cucumber.json \
--json-report --json-report-file=build/reports/report.json

Note that these two file locations are defaults for the tools in this project. You will need to modify your configuration file and the pipeline usage if you chose different ones (not fully supported yet).

gitlab-ci pipeline

In the project’s Makefile add:

include .make/xray.mk

In the projects .gitlab-ci.yml add:

- project: 'ska-telescope/templates-repository'
  file: 'gitlab-ci/includes/xray-upload.gitlab-ci.yml'

This will add a manual step to the gitlab publish stage. Note currently only the default locations for the configuration and test artefacts are used. This can’t be controlled through Makefile` variables yet.

Manual uploads

Once the test has been executed the results can also be uploaded manually using JIRA credentials.

Use –dry-run and –verbose to check for any errors first:

xray-upload -f tests/data/cucumber.json --dry-run --verbose

and then:

xray-upload -f tests/data/cucumber.json --verbose --username 'me@ad.skatelescope.org'

Tools

xray-upload

Uploads a cucumber results file to JIRA by way of the XRAY extension. Either use username/password or set the environment variable JIRA_AUTH for token authentication.

usage: xray-upload [-h] -f CUCUMBER_FILE [-u USERNAME] [-p PASSWORD]
                   [-e EXECUTION_CONFIGURATION] [-d] [-v]

options

-f, --cucumber-file

Path to the cucumber JSON results file.

-u, --username

JIRA account username

Default: “”

-p, --password

Password for the JIRA user. If not specified and the environment variable JIRA_AUTH is not set then you will be prompted for one.

Default: “”

-e, --execution-configuration

Path to test execution configuration json file (as defined by user)

Default: tests/xray-config.json

-d, --dry-run

Do not upload, just print artefacts

Default: False

-v, --verbose

Verbose output

Default: False

xray-generate-configuration

Generates a (sensible) default executionconfiguration file which describes the metadatato pass to JIRA/XRAY.

usage: xray-generate-configuration [-h] [-o OUTPUT_FILE] [-d] -c CHART_FILE
                                   [-t TEST_PLAN_DEFAULT] [-p PROJECT]

options

-o, --output-file

File name of output file

Default: tests/xray-execution.json

-d, --dry-run

Don’t write file just print output

Default: False

-c, --chart-file

Name of the chart (file) against which the tests are executed.e.g. charts/ska-example/Chart.yaml

-t, --test-plan-default

The default JIRA test plan id, e.g. ‘XTP-1234’

-p, --project

The JIRA XRAY enabled project to use

Default: “XTP”