Writing new product tests

Product tests can be written in any form that is understandable and executable by pytest. The current preferred approach is to adopt pytest-bdd. This is a pytest plugin which faciliates the use of the Gherkin standard to provide natural language descriptions of product tests. For general guidance for working with BDD tests for SKA projects see this link.

The first step in the development of a product test is to encode the behaviour that we wish to test, using the Gherkin format, into a feature file. These files should be placed in the src/ska-pss-protest/features directory, where currently implemented examples can be found. Tests should be marked on the first line using as many markers as are required to describe the test type. For example, a product test which executes a SPS pipeline which requires cuda would be tagged as follows:

@product @cuda @sps

A list of valid markers can be found by running

protest -H

or in pytest.ini. New markers may be declared in this file as required.

The test itself should be written in a separate file in src/ska-pss-protest. In order for the test to be discovered by ProTest, the file name must be in the form test_<some_description>.py. Should any new naming conventions be required, these too must be declared in pytest.ini.

To import the required BDD functionality required to write the test, import the following.

import pytest
from pytest_bdd import given, parsers, scenarios, then, when

To import ProTest, do

import ska_pss_protest

The test must then be linked to the feature file that was written in the previous step using the scenarios method as follows:

scenarios("path/to/feature/ticket.feature")

Following this, each Gherkin step described in the feature ticket can be implemented as a test function. An example of the first given step might look like this…

@given(parsers.parse("Some initial condition"))
def some_function():
     """
     Write test code
     """

As variables cannot automatically be passed between test stages (i.e., a variable declared in the given function cannot be accessed in any other functions), it is useful to define a context fixture that contains a dictionary of variables we wish to pass around the different stages, e.g.,

@pytest.fixture(scope="function")
def context():
    """
    Return dictionary containing variables
    to be shared between test stages
    """
    return {}

This dictionary can then be accessed for the purpose of added or extracting shared variables in the function declaration of the step that wishes to use it, for example,

@given(parsers.parse("Some initial condition"))
def some_function(context):
     """
     shared_variable = context["<key>"] # access existing variable

     context["<key>"] = new_variable # Create a new shared variable
     """

Accessing command line arguments

ProTest allows a number of command line arguments to provide inputs that should be shared amongst all tests that it will execute. For example, to instruct ProTest to use a set of locally stored test vectors, the user would run,

protest --cache <path/to/cache/dir> .....

ProTest passes these to the tests, where required, via conftest.py which provides a pytestconfig fixture that contains the value of the argument. This is passed to a test function in the same way as the fixture described above. For example, to access the cache directory that we pass in at the command line, we would write…

@given(parsers.parse("Some initial condition"))
def some_function(pytestconfig):
     """
     cache_dir = pytestconfig.getoption("cache")
     """

Test execution

Tests can be executed as part of test development, assuming no default parameters are overridden (i.e., that the command line arguments are set to their default values, see conftest.py) simply by running

pytest /path/to/test.py

but to ensure that they run as part of ProTest, it’s safest to update your local install of ProTest, to include your new tests. From the package root directory, run

poetry install

and then ProTest can be executed in the usual way

protest -i <marker> --cache </path/to/cache> --path </path/to/cheetah/build> --outdir </path/to/output/directory>

Detailed instruction on how to run ProTest product tests can be found in Running PSS product tests with ProTest