Integration Testing CI Pipeline

Integration tests are run in the Low PSI, via GitLab job psi-low-test.

These tests are performed at the Tango control system level, using multiple Low CBF sub-components.

Special Types of Test

Long Tests

Apply the mark @pytest.mark,long_test to any tests that take an excessively long time. To control whether these long tests will be executed, set PYTEST_MARKS.

e.g. to run tests that use real hardware but not the long tests: PYTEST_MARKS='hardware_present and not long_test'

Large Capture

To make best use of the limited number of FPGAs in the Low PSI, we sometimes use u280 FPGAs to run our CNICs. The u280 in duplex mode (used by tests) can capture ~4GB (minus overheads). If your test requires more than a u280 can hold, apply the mark @pytest.mark.large_capture to your test.

Large capture tests will be de-selected automatically if using a u280 FPGA, or you can do so explicitly if you prefer, e.g. by putting ‘not large_capture’ in PYTEST_MARKS.

Version Specification

The CI tests are designed to be able to test a particular version of any sub component(s). Versions can be requested by upstream pipelines using these environment variables:

By default, software components use their latest main-branch build, discovered automatically from the GitLab package registry (scripts/find_latest_main_sw.py queries the API and filters by pipeline.ref == "main"). If a software version contains dev, the corresponding Helm chart & container image will be sourced from GitLab instead of the SKAO Central Artefact Repository (CAR) (using a GitLab project ID specified in Makefile). An explicit release version (no dev in the string) will be sourced from CAR.

FPGA firmware defaults to the latest build, either released or built from the main branch. A specific version can be requested via the relevant <name>_FW_VERSION variable. These are always sourced from GitLab.

Firmware search behaviour can be controlled via the FW_SEARCH_MODE environment variable. Provide one or more BuildType enum names: ANY, RELEASE, MAIN, or DEV. Multiple values may be separated by comma, space, colon, or semicolon.

Hardware Selection

The Low CBF CNIC & Processor Tango devices require FPGA hardware to run.

The test-runner pod used to execute pytest needs to have access to the PCAP files created by the CNIC. This is generally achieved by running it on the same host computer.

Environment variables can be used to select the host server for all FPGA-using software (CNIC, Processor, test runner), or a different host can can be specified for the Processor devices.

  • PSI_SERVER - All CNIC, Processor, and test runner pods will run on this host server (unless CNIC_SERVER or PROC_SERVER is set). The possible values are “psi-perentie1”, “psi-perentie2”, “perentie1”, “perentie2”, “seren”. See also the files “charts/processor-<host>.yaml”, noting that the “psi-” prefix is removed from the host name.

  • PROC_SERVER - Processor devices will run on this host server, overriding PSI_SERVER.

  • PROC_FPGA - The type of FPGA hardware to use (“v80” or “u55c”, defaults to “u55c”). See also the files “charts/processor-<fpga>.yaml”.

  • PROC_NO_FLASH - Do not program V80 flash memory. This is likely to make tests fail, but it does save time…

  • CNIC_SERVER - CNIC devices and the test runner will run on this host server, overriding PSI_SERVER.

By default, one CNIC and one Processor are deployed. To use more, use these environment variables:

  • N_PROCESSORS - Number of Processors to deploy.

  • N_CNICS - Number of CNICs to deploy.

CNIC Tests

test_cnic_p4.py uses a CNIC in duplex mode & the shared PSI P4 switch to verify CNIC operation.

Processor Tests

test_processor.py shows an example of testing a Low CBF signal processing FPGA firmware image.

The example test function is test_correlator, which performs multiple Correlator tests using a requested firmware version.

The generic test routine (configure control system, transmit from CNIC, capture SUT output) is performed by the cnic_processor function.

The cnic_processor routine makes use of fixtures:

  • ready_subarray, which returns a Subarray device proxy that is, you guessed it, ready to configure.

  • cnic_and_processor, which returns device proxies for all CNICs and Processors. It configures network routing (using the Low PSI shared Connector device) between the first CNIC & first Processor. The routes are cleaned after use.

A data class ProcessorTestConfig is used to collate the various test parameters.

FPGA Register Logging

If environment variable FPGA_LOG_REGISTERS is set to either 1 or true, the cnic_processor function will ask the Processor to log register activity to /test-data/<short commit hash> <timestamp> <test name>.log.

Unique PCAP Filenames

PCAP files captured during tests are written to TEST_DATA_PATH (set via environment variable in the test-runner chart overrides). To prevent repeated or parallel CI runs from overwriting each other’s captures, each PCAP path is passed through unique_pcap_path() before use.

unique_pcap_path() infixes a unique tag into the filename stem:

  • In CI: the CI_JOB_ID is used (e.g. ptc3-subarray1.pcap → ptc3-subarray1-123456.pcap)

  • Via run-psi-test.sh: <username>-<timestamp> is used (e.g. ptc3-subarray1-jsmith-20260413-142500.pcap)

  • Running pytest directly without CI_JOB_ID set: a bare timestamp is used

It also creates the parent directory if it does not already exist.

PCAP Metadata

Alongside each PCAP file, a JSON metadata file is written with the same stem and a .meta.json extension (e.g. ptc3-subarray1-123456.meta.json). It records:

  • CNIC and Processor firmware details (version, FPGA type, etc)

  • Serial numbers of FPGA cards used

  • Some CI environment variables

  • Software component versions (from sw_versions.json bundled into the test runner)

  • UTC timestamp of the capture

Disk Space Management

Before tests run, conftest.py automatically cleans up old files under TEST_DATA_PATH to prevent the test server’s disk from filling up. This is skipped if TEST_DATA_PATH is not set (e.g. local development without hardware).

The age threshold for deletion scales with current disk usage:

Disk usage

Files older than

≤ 25% full

28 days

≤ 50% full

21 days

≤ 75% full

14 days

> 75% full

5 days

After cleanup, if less than 25% of disk space is free, the test run is aborted with an error.

File types cleaned up: bin, json, log, m, pcap, pcapng, png.

Manually Running the CI Pipeline

When developing a new test, you might want to avoid the round-trip delays of pushing to GitLab and waiting for the whole pipeline to run. Instead, you can use the run-psi-test.sh script to replicate the Low PSI test portion of the CI pipeline from the comfort of your own laptop. It accepts the same environment variables, and tries to make sensible guesses about what you’d like to do (e.g. using the latest integration container from your branch, falling back to the main branch if your branch doesn’t have one).

Extra Environment Variables for Manual Use

  • DEBUG_DIRTY: inherited from SKA Makefiles, this lets you run the tests when you have local uncommitted changes. Beware that if you have changed code in src (as opposed to tests), you need to push this to GitLab to get a new package built! If you know you have only changed tests, supply DEBUG_DIRTY=1 to run the script.

  • PYTEST_FILTER: to filter tests. For example, to run PTCs 23 and 25 PYTEST_FILTER='ptc23 or ptc25'

  • PYTEST_MARKS: to filter tests using pytest marks. For example PYTEST_MARKS='large_capture and hardware_present'

  • EXTRA_PYTEST_ARGS: as you might guess from the name, adds extra arguments to pytest. For example if you want to only collect tests EXTRA_PYTEST_ARGS=--co.