SKA MID CBF System Tests

This repository is used for BDD component-level testing of the MID CBF.

Code repository: ska-mid-cbf-system-tests

Documentation on the Developer’s portal: ReadTheDocs

Deploying the Namespace

The deployment of the namespace in the MID PSI is now controlled from the system-tests pipeline and no longer via SKAMPI.

MCS

The version of MCS is controlled via system-tests. The latest hash build versions of MCS will be deployed by default. If USE_LATEST_REL is set to true, the latest tagged version will be deployed.

To override the default behaviour in order to test a specific hash or tagged version of MCS from another branch (not main), manually set the MCS_VERSION Gitlab pipeline variable during the on_demand_psi stage. Hashes must follow the pattern: <tag>-dev.c<commit_hash> (ex. 0.3.1-dev.c5990c8b1), and tags must follow: <tag> (ex. 0.3.1).

Engineering Console (EC)

The version of EC is controlled via system-tests. The latest hash build versions of EC will be deployed by default. If USE_LATEST_REL is set to true, the latest tagged version will be deployed.

To override the default behaviour in order to test a specific hash or tagged version of EC from another branch (not main), manually set the EC_VERSION Gitlab pipeline variable during the on_demand_psi stage. Hashes must follow the pattern: <tag>-dev.c<commit_hash> (ex. 0.3.1-dev.c5990c8b1), and tags must follow: <tag> (ex. 0.3.1).

Signal Verification (SV)

The version of SV is controlled via system-tests. The latest hash build versions of SV will be deployed by default. If USE_LATEST_REL is set to true, the latest tagged version will be deployed.

To override the default behaviour in order to test a specific hash or tagged version of SV from another branch (not main), manually set the SV_VERSION Gitlab pipeline variable during the on_demand_psi stage. Hashes must follow the pattern: <tag>-dev.c<commit_hash> (ex. 0.3.1-dev.c5990c8b1), and tags must follow: <tag> (ex. 0.3.1).

Internal Schemas

Internal schemas is controlled by the tagged version in the pyproject.toml.

Testing an experimental internal-schemas hash

Add the following to the pyproject.toml:

[[tool.poetry.source]]
name = "internal-schemas-custom-source"
url = "https://gitlab.com/api/v4/projects/47018613/packages/pypi/simple/"

ska-mid-cbf-internal-schemas = { version = <tag>+dev.c<hash>, source = "internal-schemas-custom-source" }

Then execute poetry lock

Selecting Unreleased CORR and/or PST Bitstreams from Gitlab

  1. Identify the desired unreleased bitstream build in https://gitlab.com/ska-telescope/ska-mid-cbf-talondx (CORR) or https://gitlab.com/ska-telescope/ska-mid-cbf-talondx-pst (PST).

    • From the respective Pipelines page find the raw-build job from the pipeline in which the unreleased bitstream was built.

    • Copy the raw-build job ID.

  2. Create a new temporary ska-mid-cbf-engineering-console branch.

  3. Edit the https://gitlab.com/ska-telescope/ska-mid-cbf-engineering-console/-/blob/main/src/ska_mid_cbf_engineering_console/deployer/talondx_config/talondx_boardmap.json?ref_type=heads file.

  4. Under fpga_bitstreams, change the CORR and/or PST “source” from “raw” to “git”.

  5. Choose one of the following options:

    • Change the git_branch to the name of the branch containing the unreleased bitstream from step 1. i.e. “main”, “prerelease”, etc.

    • Change the job_id to the raw-build job id copied from step 1. Note: If the job_id is removed or does not exist in the talondx_boardmap.json, the bitstream from the latest successful raw-build job from the specified branch will be fetched.

  6. Push talondx_boardmap.json changes to your EC branch.

  7. In Gitlab, identify the automated EC pipeline and copy the EC version + hash from the oci-image-build job (example: 1.1.9-dev.c40fe99ca)

  8. When running a ska-mid-cbf-system-tests pipeline set the EC_VERSION to the version + hash obtained in previous step.

  9. Once all testing with unreleased bitstream is finished, delete the temporary ska-mid-cbf-engineering-console branch.

Injecting Unpublished Bitstreams or Device Servers

Notebook deployer_binary_injection.ipynb and an instructions README for performing this are located in directory ska-mid-cbf-system-tests/notebooks/deployer_binary_injection

Starting the namespace

Once you have made changes to your feature branch.

  1. Commit your changes and push them to remote.
  2. Open https://gitlab.com/ska-telescope/ska-mid-cbf-system-tests/-/pipelines
  3. A new pipeline should appear in the gitlab GUI associated to your feature branch.
  4. Click on the stage on_demand_psi (2nd stage in the pipeline). Manually click play button next to the job: psi-mid-on-demand-deploy
  5. To know which namespace is associated to your pipeline. Record the Pipeline ID associated to your pipeline in the gitlab GUI (starts with "#").
  6. On the PSI Head node (rmdskadevdu011). Run the following command: kubectl get ns
  7. Your namespace will be ci-ska-cbf-sys-tests-(PIPELINE_ID)-mid
  8. Record this namespace name if you want to run command line tests against this namespace or if you want to run kubectl commands to monitor/track the services and pods

Determining PYTEST_MARKER variable

Listings and descriptions for key test suites’ pytest markers, for usage with the PYTEST_MARKER variable, are available from the command line via command pytest --markers. For pipeline/non command line users, markers can also be found directly in the pytest.ini located in the root of the repository.

Running multiple markers from Gitlab

Pytest supports logical marker combinations (https://python-basics-tutorial.readthedocs.io/en/latest/test/pytest/markers.html#markers-together-with-and-or-not-and). To use this functionality through the Gitlab interface, surround the PYTEST_MARKER value with quotations (e.g. “speedrun1 or speedrun2”). Specifically, to run multiple markers join markers with the “or” operator.

Running/Rerunning Python Test Jobs

When running a python test job on a new namespace, it will start by initalizing devices and connected to Tango servers. At the end, when the tests are finished, adminMode will remain online to prevent powercycling. If the same job is run on the same namespace, without redeployment, it will only connect to the Tango server and then proceed to run tests, as the devices have been previously initalized. This flowchart demonstrates the structure of pytest_sessionstart, with respect to running/rerunning test jobs: Breakdown of pytest_sessionstart

Running Tests on the PSI

There are two methods for runnning the system-tests on the PSI: via gitlab or via commandline. They are described below. For more information on using the PSI, see https://confluence.skatelescope.org/display/SE/MID+PSI+Preliminary+MID+CBF+Testing

Testing your feature branch on the PSI via gitlab

  1. Open https://gitlab.com/ska-telescope/ska-mid-cbf-system-tests/-/pipelines
  2. Find your PIPELINE ID recorded from when you deployed the namespace. Under Stages, select the bubble for test. Select the edit wheel to configure the test job.
  3. Add PYTEST_MARKER (e.g., nightly4)
  4. Add TALON_LIST (e.g., 002)
  5. (Optional) add FSP_MODE_LIST (e.g. CORR or PST). Must be paired 1:1 with TALON_LIST if specified. Note: If TALON_LIST contains 4 boards or less, all boards must use the same FSP mode.
    • Note: Targets 005, 006, 007, 008 cannot be set to CORR.
  6. Set HW_CONFIG_FILE to the yaml file that matches your environment in the test_parameters/hw_config directory. Default to hw_config_psi.yaml.
  7. If using multiple VCCs or FSPs, set SLIM_FS_CONFIG_FILE to the fs_slim_*.yaml file in the test_parameters/slim_config directory. All frequency slice SLIM links are disabled if not provided, in which case only single board tests are supported.
  8. If using multiple FSPs, set SLIM_VIS_CONFIG_FILE to the vis_slim_*.yaml file in the test_parameters/slim_config directory. All visibility SLIM links are disabled if not provided, in which case only one FSP is supported.
  9. If testing a particular unreleased correlation bitstream set CORR_BITSTREAM_VERSION to the desired release version (e.g. 0.5.0). This manual setting is dependent on TARGET_SITE as manual bitstream programming does not occur with "itf" or "kapb".
    • Note: Testing of a particular released PST bitstream is not currently supported. By default, the most recent released PST bitstream will be fetched.
  10. If testing on 8-boards set SYS_PARAM to "8_boards", note that the default is "4_boards".
  11. Select "Run job"

Testing your feature branch on the PSI via commandline

INITIAL SETUP

The following steps are necessary to configure the environment for the first time. Run the following steps in an SSH terminal on the PSI head node (rmdskadevdu011.mda.ca):

git clone https://gitlab.com/ska-telescope/ska-mid-cbf-system-tests.git

#After initial clone or a submodule (e.g., ska-cicd-makefile) is updated:
	cd .../ska-mid-cbf-system-tests
	git submodule init
	git submodule update
		
#Run the following commands to setup your python virtual environment so that the pytests can be executed:
	python3.10 -m venv venv
	source venv/bin/activate
	python -m pip install poetry==2.1.3
	poetry install

SUBSEQUENT TESTING

Once the environment has been set up, only the following commands need to be executed.

cd .../ska-mid-cbf-system-tests
source venv/bin/activate
make python-test KUBE_NAMESPACE=... PYTEST_MARKER=... TALON_LIST=...

Note the KUBE_NAMESPACE is the namespace recorded when following the deployment instructions above

Note that there are other options for python-test as well.

Running tests at the ITF site

Tests can be run on the ITF by using the TARGET_SITE variable and specifying itf, which must be done at the very beginning of the new pipeline to ensure the correct runner is used. This will cause the Makefile to automatically use ITF visibilities pod file (visibilities_pod_itf.yaml), as well as deploying a namespace in the ITF environment. However currently the ITF hardware file must be specified during the test execution (hw_config_itf.yaml). Note that deleting such a namespace can be difficult if something goes wrong in the pipeline, requiring VPN access and permissions at the ITF.

Differences between the ITF and the PSI runs include:

  • ITF automatic namespace deletion stage is currently failing

    • Manual deletion of the namespace is necessary and cannot be done from Gitlab, so ensure you have the ability to manually delete a namespace at the ITF before creating one

  • QSPI is not updated at ITF

  • bitstream version is not updated in deployer pod at ITF

  • visibilities are not saved in the system-tests-artifacts namespace (no ITF equivalent)

Running tests at the KAPB site

Tests run at the KAPB site are not run through the Gitlab pipeline. Instead, the instructions here should be followed on Confluence here.

Note that the KAPB hw config file does not and should not include the LRU username and password for security reasons. Therefore the LRUs cannot be checked directly the way those in the PSI are.

Differences between the KAPB and PSI runs include:

  • KAPB runs cannot be run on Gitlab, must be run manually

  • hw config file is downloaded from vault to cbf controller pod and is not overwritten by local copy

    • local copy of hw config file does not include LRU password, which should not be made public, LRU status cannot be directly checked

      • MCS On test cannot check whether LRUs are on

      • (sidenote: adding the lru password locally was tried and accessing the KAPB LRUs through the powerswitch python functions still errored out on a permission error)

    • local hw config exists for reference and for pinging talons from system tests

      • changing the local hw config will not overwrite the file in the cbfcontroller

      • if KAPB hw config is updated, the local copy will need to be re-copied from the cbf controller pod and lru password removed before committing to git

  • QSPI is not updated at KAPB

  • bitstream version is not updated in deployer pod at KAPB

  • visibilities are not saved in the system-tests-artifacts namespace (no KAPB equivalent)

  • slim config files are automatically downloaded from the vault to the cbfcontroller for KAPB and do not need to be specified at runtime

    • the slim config files are copied from the cbfcontroller in sessionstart to the slim config folder as the DCT checkpoint uses them

Pylint

Note that some of the lint issues cannot be resolved due to the nature of the BDD tests and those errors have been added to the disabled list when pylint is run in the pipeline.

Run the following command locally to make sure your code will pass the lint test in the pipeline:

cd .../ska-mid-cbf-system-tests
source venv/bin/activate
make python-lint

Notes

Even though the receptors to be released can be specified in release_receptors.json, currently, all the receptors will be released at the beginning and end of each test as part of subarray_to_empty_state() call.

Test Order and Delay Model Interaction

Delay model tests utilizing the same non-reset subarray require that each delay model advances in time w.r.t their start time, or it will be ignored. This means that a certain test order is required for tests to work as intended, which is especially important in the nightly markers. To current understanding pytest determines this test order via alphabetical order of the test .py files, then top to bottom order of test function definitions in the .py file. Delay model tests must be labelled and placed according to this order to ensure that they work when using collecting markers like nightly4. For a future quick more permanent fix to remove this alphabetical ordering to delay model test functionality, nightly order can be determined explicitly by including order metadata on the nightly marks and sorting tests in the conftest using hook pytest_collection_modifyitems.

Timeouts

If a system-tests job fails due to a job exceeding a duration timeout, the shorter of the following two timeout parameters has been reached and must be increased:

  • The system-tests project CI/CD timeout, found in Settings -> CI/CD -> General Pipelines.

  • The Gitlab ska-psi-mid-runner timeout set by the systems team (this can be requested by creating an STS ticket). It is recommended to keep both timeout parameters in sync and beyond the expected maximum duration of a python-test job.