Integration Testing CI Pipeline =============================== Integration tests are run in the Low PSI, via GitLab job ``psi-low-test``. These tests are performed at the Tango control system level, using multiple Low CBF sub-components. Special Types of Test --------------------- Long Tests ^^^^^^^^^^ Apply the mark ``@pytest.mark,long_test`` to any tests that take an excessively long time. To control whether these long tests will be executed, set ``PYTEST_MARKS``. e.g. to run tests that use real hardware but not the long tests: ``PYTEST_MARKS='hardware_present and not long_test'`` Large Capture ^^^^^^^^^^^^^ To make best use of the limited number of FPGAs in the Low PSI, we sometimes use u280 FPGAs to run our CNICs. The u280 in duplex mode (used by tests) can capture ~4GB (minus overheads). If your test requires more than a u280 can hold, apply the mark ``@pytest.mark.large_capture`` to your test. Large capture tests will be de-selected automatically if using a u280 FPGA, or you can do so explicitly if you prefer, e.g. by putting 'not large_capture' in ``PYTEST_MARKS``. Version Specification --------------------- The CI tests are designed to be able to test a particular version of any sub component(s). Versions can be requested by upstream pipelines using these environment variables: - ``CNIC_FW_VERSION``: `CNIC Firmware `_ - ``CORR_FW_VERSION``: `Correlator Firmware `_ - ``PSS_FW_VERSION``: `PSS Firmware `_ - ``PST_FW_VERSION``: `PST Firmware `_ - ``SKA_LOW_CBF_VERSION``: `Low CBF Monitoring & Control `_ - ``SKA_LOW_CBF_PROC_VERSION``: `Processor Tango Device `_ - ``SKA_LOW_CBF_TANGO_CNIC_VERSION``: `CNIC Tango Device `_ By default, software components use their **latest main-branch build**, discovered automatically from the GitLab package registry (``scripts/find_latest_main_sw.py`` queries the API and filters by ``pipeline.ref == "main"``). If a software version contains ``dev``, the corresponding Helm chart & container image will be sourced from GitLab instead of the `SKAO Central Artefact Repository (CAR) `_ (using a GitLab project ID specified in ``Makefile``). An explicit release version (no ``dev`` in the string) will be sourced from CAR. FPGA firmware defaults to the latest build, either released or built from the main branch. A specific version can be requested via the relevant ``_FW_VERSION`` variable. These are always sourced from GitLab. Firmware search behaviour can be controlled via the ``FW_SEARCH_MODE`` environment variable. Provide one or more ``BuildType`` enum names: ``ANY``, ``RELEASE``, ``MAIN``, or ``DEV``. Multiple values may be separated by comma, space, colon, or semicolon. Hardware Selection ------------------ The Low CBF CNIC & Processor Tango devices require FPGA hardware to run. The `test-runner` pod used to execute pytest needs to have access to the PCAP files created by the CNIC. This is generally achieved by running it on the same host computer. Environment variables can be used to select the host server for all FPGA-using software (CNIC, Processor, test runner), or a different host can can be specified for the Processor devices. - `PSI_SERVER` - All CNIC, Processor, and test runner pods will run on this host server (unless `CNIC_SERVER` or `PROC_SERVER` is set). The possible values are "psi-perentie1", "psi-perentie2", "perentie1", "perentie2", "seren". See also the files "charts/processor-.yaml", noting that the "psi-" prefix is removed from the host name. - `PROC_SERVER` - Processor devices will run on this host server, overriding `PSI_SERVER`. - `PROC_FPGA` - The type of FPGA hardware to use ("v80" or "u55c", defaults to "u55c"). See also the files "charts/processor-.yaml". - `PROC_NO_FLASH` - Do not program V80 flash memory. This is likely to make tests fail, but it does save time... - `CNIC_SERVER` - CNIC devices and the test runner will run on this host server, overriding `PSI_SERVER`. By default, one CNIC and one Processor are deployed. To use more, use these environment variables: - `N_PROCESSORS` - Number of Processors to deploy. - `N_CNICS` - Number of CNICs to deploy. CNIC Tests ---------- ``test_cnic_p4.py`` uses a CNIC in duplex mode & the shared PSI P4 switch to verify CNIC operation. Processor Tests --------------- ``test_processor.py`` shows an example of testing a Low CBF signal processing FPGA firmware image. The example test function is ``test_correlator``, which performs multiple Correlator tests using a requested firmware version. The generic test routine (configure control system, transmit from CNIC, capture SUT output) is performed by the ``cnic_processor`` function. The ``cnic_processor`` routine makes use of fixtures: - ``ready_subarray``, which returns a Subarray device proxy that is, you guessed it, ready to configure. - ``cnic_and_processor``, which returns device proxies for all CNICs and Processors. It configures network routing (using the Low PSI shared Connector device) between the first CNIC & first Processor. The routes are cleaned after use. A data class ``ProcessorTestConfig`` is used to collate the various test parameters. FPGA Register Logging ^^^^^^^^^^^^^^^^^^^^^ If environment variable ``FPGA_LOG_REGISTERS`` is set to either ``1`` or ``true``, the ``cnic_processor`` function will ask the Processor to log register activity to ``/test-data/ .log``. Unique PCAP Filenames ^^^^^^^^^^^^^^^^^^^^^ PCAP files captured during tests are written to ``TEST_DATA_PATH`` (set via environment variable in the test-runner chart overrides). To prevent repeated or parallel CI runs from overwriting each other's captures, each PCAP path is passed through ``unique_pcap_path()`` before use. ``unique_pcap_path()`` infixes a unique tag into the filename stem: - In CI: the ``CI_JOB_ID`` is used (e.g. ``ptc3-subarray1.pcap`` → ``ptc3-subarray1-123456.pcap``) - Via ``run-psi-test.sh``: ``-`` is used (e.g. ``ptc3-subarray1-jsmith-20260413-142500.pcap``) - Running pytest directly without ``CI_JOB_ID`` set: a bare timestamp is used It also creates the parent directory if it does not already exist. PCAP Metadata ^^^^^^^^^^^^^ Alongside each PCAP file, a JSON metadata file is written with the same stem and a ``.meta.json`` extension (e.g. ``ptc3-subarray1-123456.meta.json``). It records: - CNIC and Processor firmware details (version, FPGA type, etc) - Serial numbers of FPGA cards used - Some CI environment variables - Software component versions (from ``sw_versions.json`` bundled into the test runner) - UTC timestamp of the capture Disk Space Management --------------------- Before tests run, ``conftest.py`` automatically cleans up old files under ``TEST_DATA_PATH`` to prevent the test server's disk from filling up. This is skipped if ``TEST_DATA_PATH`` is not set (e.g. local development without hardware). The age threshold for deletion scales with current disk usage: +--------------------+-----------------+ | Disk usage | Files older than| +====================+=================+ | ≤ 25% full | 28 days | +--------------------+-----------------+ | ≤ 50% full | 21 days | +--------------------+-----------------+ | ≤ 75% full | 14 days | +--------------------+-----------------+ | > 75% full | 5 days | +--------------------+-----------------+ After cleanup, if less than 25% of disk space is free, the test run is aborted with an error. File types cleaned up: ``bin``, ``json``, ``log``, ``m``, ``pcap``, ``pcapng``, ``png``. Manually Running the CI Pipeline -------------------------------- When developing a new test, you might want to avoid the round-trip delays of pushing to GitLab and waiting for the whole pipeline to run. Instead, you can use the `run-psi-test.sh` script to replicate the Low PSI test portion of the CI pipeline from the comfort of your own laptop. It accepts the same environment variables, and tries to make sensible guesses about what you'd like to do (e.g. using the latest integration container from your branch, falling back to the main branch if your branch doesn't have one). Extra Environment Variables for Manual Use ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - ``DEBUG_DIRTY``: inherited from SKA Makefiles, this lets you run the tests when you have local uncommitted changes. Beware that if you have changed code in ``src`` (as opposed to ``tests``), you need to push this to GitLab to get a new package built! If you know you have only changed tests, supply ``DEBUG_DIRTY=1`` to run the script. - ``PYTEST_FILTER``: to filter tests. For example, to run PTCs 23 and 25 ``PYTEST_FILTER='ptc23 or ptc25'`` - ``PYTEST_MARKS``: to filter tests using pytest marks. For example ``PYTEST_MARKS='large_capture and hardware_present'`` - ``EXTRA_PYTEST_ARGS``: as you might guess from the name, adds extra arguments to ``pytest``. For example if you want to only collect tests ``EXTRA_PYTEST_ARGS=--co``.