Getting Started

This project provides command-line tools to configure the JIRA/XRAY Test Execution and manage the upload of test results.

Supported test results are of the form:

Cucumber JSON (generated through pytest-bdd)
Robot Framework reports (output.xml)
Pytest (using the JUnit XML format)

There are two tools:

xray-generate-configuration helps you to configure your package, so that the right BDD test results are uploaded to the right place with the right metadata. It generates a test execution configuration file, i.e. tests/xray-config.json. You will probably only ever run this tool once, to bootstrap an initial configuration file. After that, modifications to the configuration can be effected by manually editing the configuration file.
xray-upload performs the upload of BDD test results to Jira Xray.

Installing

How you install this package depends on how you intend to use it. Choose from one of these two use cases:

Developers will routinely upload test results as part of their development workflow. They need ongoing access to the xray-upload tool. In this case, you will want this package to be one of your package’s development dependencies, so that the xray-upload is available as part of your package’s development installation. If using poetry, for example, you would
```
me@local:~$ poetry add --group dev ska-ser-xray
```
Uploading of test results will be performed automatically in the CI/CD pipeline, rather than manually by developers. Developers do not need access to the xray-upload tool. In that case, you only need to install this package temporarily, in order to run the one-off xray-generate-configuration tool, and then you can uninstall it again:
```
me@local:~$ pip install ska-ser-xray
me@local:~$ # use the xray-generate-configuration tool to bootstrap your configuration
me@local:~$ pip uninstall ska-ser-xray
```

Configuration

When using the functionality for the first time it is necessary to create a configuration file which provides metadata for the XRAY Test Execution. It has defaults and can be customised through environment variables.

In the example below it can be interpreted as:

for the field name (line 1)
read the environment variable CI_COMMIT_BRANCH (line 2)
if it exists - use its value
if it doesn’t - use default as the value
look up the corresponding key in the definitions (line 4-5)

So if it isn’t found in the environment variables, the XRAY configuration will be:

name="branch execution"

example-config.json section

  "name": {
    "env": "CI_COMMIT_BRANCH",
    "maps_to": {
      "default": "branch execution",
      "main": "main branch execution"
    }

A configuration file can be generated through xray-generate-configuration. There are three configuration items which should be provided to the command:

–project the XRAY enabled JIRA project to use
–chart-file the relative path to the Chart.yaml of the repository’s helm chart used for the BDD tests.
–test-plan-default a XRAY test plan issue id to link to by default. This is required to be of JIRA issue type Test Plan!

Example:

xray-generate-configuration -c charts/ska-example/Chart.yaml -p XTP -t "XTP-0000"

which creates tests/xray-config.json. This file needs to be committed. Values where the defaults weren’t provided on the command-line or didn’t resolve are set to %DEFINE_ME%. These will need to be replaced manually before a successful upload can happen.

Advanced configuration

For a custom project other than XTP there needs to be a little more configuration. In the config file (xray-config.json), custom projects as well as the settings for fields can be addressed. The example below modifies:

The hostname jira_url
The XRAY enabled project project.id
For the customfield_mapping
- start_date and end_date don’t exist in the project
- test_environments and test_plans have specific customfield_<n> mappings

"jira_url": "https://jira-test.skatelescope.org",
"project": {
   "id": "TE",
   "issuetype_id": 11602,
   "customfield_mapping": {
      "start_date": "",
      "end_date": "",
      "test_plans": "customfield_11927",
      "test_environments": "customfield_11925"
   }
}

Note

For all https://jira.skatelescope.org hosted projects only the project.id needs to be changed from the generated defaults.

Usage

Once a configuration has been settled on, the upload functionality can be used in the project’s CI/CD pipeline, or manually.

Uploading Cucumber test results

When using Cucumber, test results to be uploaded will be generated by two pytest plugins:

pytest-bdd allows you to run BDD tests using pytest, and produces BDD test results in cucumber.json format
pytest-json-report produces test results in report.json format.

These two plugin packages must be installed in the environment in which you run your tests, e.g. your package’s development environment.

The pytest command needs to have the following extra arguments:

--cucumberjson=build/reports/cucumber.json \
--json-report --json-report-file=build/reports/report.json

Then, update your project’s Makefile to include:

include .make/xray.mk

XRAY_TEST_RESULT_FILE ?= build/reports/cucumber.json
XRAY_EXECUTION_CONFIG_FILE ?= tests/xray-config.json

Uploading Robot Framework test results

When using Robot Framework, test results to be uploaded will be generated automatically by the robot command.

Robot Framework outputs multiple report files, but only output.xml should be uploaded to Xray.

Note

Robot Framework generates the report files in the working directory by default. This can be changed by using the –outputdir option, for instance:

robot --outputdir build/robot path/to/tests

Then, update your project’s Makefile to include:

include .make/xray.mk

XRAY_EXTRA_OPTS ?= -t robot_framework
XRAY_TEST_RESULT_FILE ?= build/robot/output.xml
XRAY_EXECUTION_CONFIG_FILE ?= tests/xray-config.json

Uploading Pytest results

When using Pytest, the test results to be uploaded are generated when pytest is invoked with the option to output JUnit XML results:

--junitxml=build/reports/pytest-junit.xml

First, follow Adding Xray metadata to Pytest.

Then, update your project’s Makefile to include:

include .make/xray.mk

XRAY_EXTRA_OPTS ?= -t pytest
XRAY_TEST_RESULT_FILE ?= build/reports/pytest-junit.xml
XRAY_EXECUTION_CONFIG_FILE ?= tests/xray-config.json

GitLab CI/CD configuration

It is recommended to upload test results to Xray from a GitLab CI/CD pipeline.

In the project’s .gitlab-ci.yml add:

include:
  - project: 'ska-telescope/templates-repository'
    file: 'gitlab-ci/includes/xray-publish.gitlab-ci.yml'

(If the include: key is already present, append this new include to the existing list of includes.)

This will add a xray-publish job to the publish stage in your pipeline.

Note

To prevent Xray from being polluted with Test Executions for each CI/CD pipeline, the xray-publish job is set to never run automatically.

The job can be triggered manually from the CI/CD pipeline UI in GitLab.

Note

There is no need to provide Jira credentials in your CI pipeline configuration; these are already set in the pipeline environment.

Manual uploads

Once the test has been executed the results can also be uploaded manually using JIRA credentials.

Use –dry-run and –verbose to check for any errors first:

xray-upload -f tests/data/cucumber.json --dry-run --verbose

and then:

xray-upload -f tests/data/cucumber.json --verbose --username 'me@ad.skatelescope.org'

Tools

xray-upload

Uploads a test results file to JIRA by way of the XRAY extension. Either use username/password or set the environment variable JIRA_AUTH for token authentication.

usage: xray-upload [-h] [-t {cucumber,robot_framework,pytest}] -f FILE
                   [-u USERNAME] [-p PASSWORD] [-e EXECUTION_CONFIGURATION]
                   [-d] [-v]

Named Arguments

-t, --test-framework

Possible choices: cucumber, robot_framework, pytest

The test framework that generated the results

Default: “cucumber”

-f, --file

Path to the test results file.

-u, --username

JIRA account username

Default: “”

-p, --password

Password for the JIRA user. If not specified and the environment variable JIRA_AUTH is not set then you will be prompted for one.

Default: “”

-e, --execution-configuration

Path to test execution configuration json file (as defined by user)

Default: tests/xray-config.json

-d, --dry-run

Do not upload, just print artefacts

Default: False

-v, --verbose

Verbose output

Default: False

xray-generate-configuration

Generates a (sensible) default executionconfiguration file which describes the metadatato pass to JIRA/XRAY.

usage: xray-generate-configuration [-h] [-o OUTPUT_FILE] [-d] -c CHART_FILE
                                   [-t TEST_PLAN_DEFAULT] [-p PROJECT]

Named Arguments

-o, --output-file

File name of output file

Default: tests/xray-config.json

-d, --dry-run

Don’t write file just print output

Default: False

-c, --chart-file

Name of the chart (file) against which the tests are executed.e.g. charts/ska-example/Chart.yaml

-t, --test-plan-default

The default JIRA test plan id, e.g. ‘XTP-1234’

-p, --project

The JIRA XRAY enabled project to use

Default: “XTP”