SKA Mid.CBF I&T Tests

Description

This internal CIPA repository primarily contains LMC emulation functionality to replicate LMC control and interaction with CBF through MCS and secondarily contains supporting tools to facilitate, monitor, and test such access. LMC emulation is driven through pytest automated tests or manual Jupyter notebooks for use cases of Mid.CBF integration testing, developer prototyping, and bug fixing. Integration milestones M3, M2, and H2, which are largely driven by issuing commands to MCS, are contained in this repository.

Code repository: here

ReadtheDocs: here

Integration Milestones

Integration milestones are mentioned throughout the README and code base, please see 311-200000-045 Mid.CBF Detailed I&T Plan for the definition of the milestones.

Table Of Contents

Guides
Technical Notes

Guides and Information

Setup I&T Tests

Clone the repository git clone https://gitlab.com/ska-telescope/ska-mid-cbf-int-tests.git
Create a virtual environment python3.10 -m venv <path-ending-in-dir-of-venv-name>
Activate the virtual environment source <path-ending-in-dir-of-venv-name>/bin/activate
Install poetry pip install poetry
Navigate into the cloned repository
Install dependencies to the venv poetry install

Using Namespaces

Creating a namespace:

Open the Gitlab repository in a browser
Navigate to Pipelines
Click New pipeline, and identify the newly created pipeline associated with your username
Wait for initial jobs to run
Select the trigger-downstream-deployment
Decide on dependency versions to deploy:
- To use the tagged versions specified in ska-mid-cbf-deployment/charts from CAR set DEV_DEPLOY=false
- To use the latest dev hash versions from respective Gitlab repos, set DEV_DEPLOY=true
  - Ability to specific dev versions of MCS, FHS-VCC, FHS-FSP, etc coming soon.
Click run on the job, once completed this will create a namespace for you, the name of the namespace created will be viewable in the job log.

Redeploying a namespace: run the k8s-on-demand-redeploy job from the downstream ska-mid-cbf-deployment pipeline.

Destroying a namespace: either run the k8s-on-demand-destroy job from the downstream ska-mid-cbf-deployment pipeline or run command kubectl delete namespace <name-of-namespace> on a node connected to the relevant cluster. Destruction will be automatically run if the python-test job is run on Gitlab with job <site>-mid-on-demand-destroy in the stage cleanup.

Services

Test tool service modules are implemented within int-tests src directory and are used by the tests. Such services are tested under the services_test marker and are listed below with a short description on their primary function:

Services:

test_observer: The test_observer service provides the object TestObserver which can be set in TestObserverMode.NOT_ASSERTING mode or TestObserverMode.ASSERTING mode, and respectively either logs observations the observer makes of given values or logs and asserts on the given values in line moreso with a test.

System Under Test Diagrams

Simulator Deployment	Emulator Deployment	Hardware Deployment

Automated Testing

The following GitLab CI variables can be set when running a pipeline. If not set the values in the default column below are used.

Variable	Description	Default value	Options
MILESTONE	The milestone directory containing MILESTONE_CONFIG_FILE in ska-mid-cbf-deployment/config/deployment_config	dev	see options in MILESTONE variable of `.gitlab-ci.yaml`
MILESTONE_CONFIG_FILE	The milestone configuration file defining the deployment by ska-mid-cbf-deployment, yaml configuration file in ska-mid-cbf-deployment/config/deployment_configs/MILESTONE	psi_sim_2v_1f.yaml	see options in MILESTONE_CONFIG_FILE variable of `.gitlab-ci.yaml`
DEV_DEPLOY	Determines if charts use latest commit hash on main branches (true) or latest tagged releases in CAR (false)	true	true false
RUNNER_TAG	The GitLab runner tag to be used	k8srunner-psi-mid	see options in RUNNER_TAG variable of `.gitlab-ci.yaml`
SYS_PARAM	Index of system parameters to be used from the init_sys_param.json	36_vccs_same_k	see options in SYS_PARAM variable of `.gitlab-ci.yaml`
PYTEST_MARKER	Selects which tests to run.	client_test	see options in `pytest.ini`

Running automated testing:

Run the steps to create a namespace
- For running on Gitlab:
  - Click into the python-test job in the test stage
  - Fill the variables in with the above listed required parameters
  - Run the job
- For running locally:
  - Ensure you’re running on a machine that can access the relevant kubernetes cluster of the created namespace
  - Navigate to your local code repository
  - Run command make python-test KUBE_NAMESPACE=<namespace-name> KEYWORD1=VALUE1 KEYWORD2=VALUE2 ... replacing <namespace-name> with the name of the created namespace and KEYWORDn, VALUEn pairs with the above required parameters

Prototyping Notebook

Running the notebook:

Run the steps to create a namespace
Ensure you’re running on a machine that can access the relevant kubernetes cluster of the created namespace
Navigate to ska-mid-cbf-int-tests/notebooks/prototyping
Fill in the parameters in notebook_parameters.json, explanation of the parameters is located in the accompanying notebook_params_README.md
Begin running an ipython kernel with command jupyter-notebook --no-browser
Open the notebooks in notebooks/prototyping
Run the notebooks going in index order and following listed instructions in notebook if any

Cleaning up the notebook to upload to Gitlab: in the local code repo run ska-mid-cbf-int-tests/scripts/notebooks/clear-notebooks-data.sh to clear outputs and metadata on all notebooks in the notebooks directory

Log Collection

Log collection is automatically run in the Gitlab pipeline after the test stage python-test job.

To run manually:

Ensure a namespace was created (see Creating a Namespace in Using Namespaces)
Navigate to directory ska-mid-cbf-int-tests/scripts/log_collection
Run ./collect_namespace_pod_logs.sh <name-of-namespace>
Run ./summarize_pod_images.sh <name-of-namespace>
Run ./summarize_log_levels.sh
Logs should be generated in ska-mid-cbf-int-tests/logs

Using Dev Hash versions of Common Test Infrastructure

All python dependencies built correctly by the SKA build system and accessed through this repository’s pyproject.toml can be accessed at a given dev hash version directly from that dependency’s Gitlab package repository, instead of via a tagged release from CAR. Specifically, this can be utilized to use and test development branches of common-test-infrastructure.

To do so in the pyproject.toml:

Uncomment the “gitlab-pkg-registry-common-test-infrastructure” source
Replace the tag value given to ska-mid-cbf-common-test-infrastructure with { version = <desired dev hash>, source = "gitlab-pkg-registry-common-test-infrastructure" }
Run poetry lock

Design Principles

LMC Emulation for MCS Interaction

Code in ska_mid_cbf_int_tests should seek to emulate LMC as much as possible through its interactions with MCS. See https://developer.skao.int/projects/ska-mid-cbf-mcs/en/latest/guide/interfaces/lmc_mcs_interface.html for the expected interface. This is important to do as replicating LMC behavior increases the accuracy of tests and prototyping to its final product usage. This is also helpful in ensuring that M level and S level test code can both be handled through the same command infrastructure, as code from both milestone classes following the LMC emulation principle would avoid major differences, and from this such milestones could avoid many major conflicts arising from such differences. An example of the principle is in the SubarrayClient in ska_mid_cbf_int_tests/cbf_command which replicates commands in the LMC-MCS interface directly and only additionally contains subroutines that would be possible sequences for LMC to run.

Minimal MCS Sequence Paths

Authoring and infrastructure should be designed to reduce the number of MCS command sequence paths as much as possible, with the ideal for all tests to be utilizing a small suite of singular invariant non-branching sequences of MCS commands. Fewer sequences reduces the chance of stray and conflict between milestones, and between tests within a milestone, and protect against having an unmanagable number of cases of nearly duplicated code to maintain, which MCS command sequences (particularly scans) are likely to represent.

Technical Notes

Data Access Through importlib.resources

The paradigm for data access in ska_mid_cbf_int_tests is to use the modern Python library of importlib.resources (python >= 3.7) to import data through Python’s import builtin system. This allows data to be imported anywhere in the repository as long as the project is built and installed (via poetry) and corresponding data is included in that build. This is necessary such that data can be commonly used in both the testing executions and also the notebooks executions without tying to specific common relative paths which would introduce high coupling to the data location. See https://docs.python.org/3/library/importlib.resources.html for further information and usage.

This means data must be kept in the build directories (src directory) and included in the installed package of the repository which is also under the name ska_mid_cbf_int_tests. Data must be registered in the build specification to be included in the ska_mid_cbf_int_tests package build, as only .py files are included default. Modify the pyproject.toml “include” variable to include non py files as per https://python-poetry.org/docs/pyproject/#exclude-and-include.

Docstring Style and Sphinx API Auto-Documentation

The docstring style of ska_mid_cbf_int_tests is sphinx https://sphinx-rtd-tutorial.readthedocs.io/en/latest/docstrings.html. This is necessary for automatic API documentation using the sphinx autodoc extension.

Reference https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html to see how to setup autodoc, such that it automatically generates API documentation from API docstrings in the generated ReadtheDocs pages. Modify files in docs as appropriate to implement this. Importantly, two things to be aware of:

Documentation for modules may just generate nothing which commonly is caused by the module not being importable to docs generation code. This requires modifying sys.path attributes in conf.py in the docs directory to get the module importable and recognized.
API to be documented requires all of their dependencies installed since they are imported as normal modules for documentation generation, so ensure that the docs dependencies group includes all normal dependencies of the API.

Downstream ska-mid-cbf-deployment Pipelines

All deployment related activities are managed via ska-mid-cbf-deployment. The ska-mid-cbf-int-tests deploy-trigger-downstream stage triggers a ska-mid-cbf-deployment pipeline to deploy the system under test as defined in the MILESTONE_CONFIG_FILE to the corresonding MILESTONE_CONFIG_FILE. The ska-mid-cbf-int-test pipeline will wait until completion of the ska-mid-cbf-deployment pipeline and then executes a get-deployment-pipeline-id job which runs a script to fetch the downstream deployment pipeline ID needed to form the ska-mid-cbf-deployment namespace name.

MCS Version and Schemas for client_test Policy

Use latest MCS version via DEV_DEPLOY='true' option for client_tests. Schema versions used in client_test can be updated to newer schema when useful (e.g. ensuring a schema is valid for an associated milestone in int-tests) or when currently used schema is deprecated, however maintaining latest schema version is not necessary as point of client_test is to test int-tests MCS client infrastructure interaction with MCS, and not unit testing MCS’ support of latest schema (this is to be handled on MCS side).