.. _testing: Automated integration tests =========================== Here, you will find information on how tests in ska-sdp-integration are implemented, including what you need to know if you want to add your own tests. We also describe how you can run the tests in different environments and give an overview of existing automated tests. The SDP integration tests aim to test the interactions between various components in a real-life Kubernetes set up. SDP is deployed on its own with all of its components, which then interact with each other without any of them being mocked. These include tests of the controller and subarray devices, as well as various processing script and pipeline tests (e.g. vis-receive, pointing-offset, etc.). The tests and all related files can be found in the `tests directory `__. All of the file paths given below are relative to this directory. ``context`` fixture ------------------- ``context`` is a pytest fixture defined in ``tests/conftest.py``. It contains variables and definitions that are global to all tests and are used by most of the tests. The context object is split into groups of related parameters loaded as sub-objects: - SDP - Namespace and subarray information loaded from environment variables - Controller, subarray and queue connector device names - QA metrics and Kafka hosts - Timeouts - Timeout values loaded from environment variables Use this fixture if you need to add new information similar to the above specifications. Environment variables ^^^^^^^^^^^^^^^^^^^^^ The following variables are included in the SDP context sub-object and used in the tests. In this table, default indicates the value loaded into tests, if the environment variable is not specified: .. list-table:: :widths: auto :header-rows: 1 * - Name - Description - Default * - ``KUBE_NAMESPACE`` - Namespace where the control system is running - ``None`` * - ``SUBARRAY_ID`` - Which Subarray Tango device to use for the test (e.g. test-sdp/subarray/01) - ``01`` In addition, various custom timeout variables are also used, see below in section “Custom timeouts”. Note that there are also some environment variables that are only used to set up the Tango client, and these are specified directly in ``tests/common/tango/__init__.py`` .. list-table:: :widths: auto :header-rows: 1 * - Name - Description - Default * - ``TEST_INGRESS`` - Ingress to connect to the cluster - if not set, the test is assumed to be running inside the cluster - ``None`` * - ``TEST_TANGO_CLIENT`` - Which tango client to use: dp or gql - ``dp`` * - ``TARANTA_AUTH_URL`` - URL of Taranta authentication service, needed when using TangoGQL client - ``None`` * - ``TARANTA_USERNAME`` - Username to get cookies from Taranta authentication service, needed when using TangoGQL client - ``None`` * - ``TARANTA_PASSWORD`` - Password to get cookies from Taranta authentication service, needed when using TangoGQL client - ``None`` The following environment variable is used inside ``tests/common/kubernetes.py`` to determine whether “helm uninstall” supports –wait: .. list-table:: :widths: auto :header-rows: 1 * - Name - Description - Default * - ``HELM_UNINSTALL_HAS_WAIT`` - Does ``helm uninstall`` support ``--wait``? Yes only if set to ``1`` - ``None`` And the TEST_MARKER environment variable is used in the Makefile to set which tests will run (see later): .. list-table:: :widths: auto :header-rows: 1 * - Name - Description - Default * - ``TEST_MARKER`` - Only run BDD tests with this tag (or a combination of tags using ``and`` and ``not``) - ``None`` Custom timeouts ^^^^^^^^^^^^^^^ Some of the command and test execution is controlled by timeouts. These make sure that a step waits for a reasonable amount of time for the execution of a long-running command or a long-running test step. The most important timeouts can be controlled via environment variables, which are loaded into the ``context`` fixture via the ``timeouts`` sub-object. All of the values are given in seconds: .. list-table:: :widths: auto :header-rows: 1 * - Name - Description - Default * - ``TEST_TIMEOUT_WAIT_POD`` - Timeout for waiting for a pod to start (including downloading images) - ``300`` * - ``TEST_TIMEOUT_DEVICE_ON_OFF`` - Timeout for the QueueConnector and MockDish devices to change state to ON or OFF - ``20`` * - ``TEST_TIMEOUT_WAIT_TO_EXECUTE`` - Wait for various resources, image downloads, some processing; for QA tests - ``180`` When adding a new test, please make sure you either use these, or add a new timeout, if relevant, instead of hard-coding the values in your test. Commands other than AssignResources and Configure are not long-running, hence the timeout waiting for the state-transition as a result of these commands cannot be controlled and is set to 60 seconds as a default. ``telescope_metadata`` fixture ------------------------------ ``telescope_metadata`` is a pytest fixture defined in ``tests/integration/conftest.py`` as part of a shared ``given`` step. It initializes an instance of the class ``MetadataConfiguration`` with the given ``telescope_name``: .. code-block:: python @given( parsers.parse("I select the dataset for the {telescope_name:S} telescope"), target_fixture="telescope_metadata", ) def set_config_params(telescope_name): """ Set parameters to determine test configuration """ config = MetadataConfiguration(telescope_name) return config The ``telescope_name`` is used to select which data and telescope metadata a test should use. This means that any test can specify which data it wants to use by including the line ``I select the dataset for the telescope`` in the feature file. Use the telescope name ‘Mid’ and ‘Low’ (not case-sensitive) to choose the desired configuration. The ``MetadataConfiguration`` configuration class sets any parameters which vary between test scenarios. Values to construct these variables and the ``assign_resources`` string are stored in a separate json file called ``telescope_metadata.json``. Data and telescope metadata ^^^^^^^^^^^^^^^^^^^^^^^^^^^ The data used in the integration tests is located at ``tests/resources/data`` under ``AA05LOW.ms`` for Low and ``pointing-data/scan-1.ms`` for Mid. The latter is saved as a .tar file together with the remaining 4 pointing MS and pointing HDF files. Additional datasets can be added here for new test cases. All 5 MeasurementSets and pointing HDF files are used for the ``pointing`` pipeline tests. Note, that if a test uses data from ``tests/resources/data/pointing-data``, they need to set the ``"data_tar_file": "pointing-data.tar"`` key in the telescope_metadata.json file. This makes sure the data is extracted from the .tar file after it is copied to the persistent volume claim. Using the ``data_tar_file`` key allows you to provide ``.tar`` files with data for testing. Shared BDD steps ---------------- Shared BDD steps are located in ``tests/integration/conftest.py``. Common pytest fixtures are defined in ``tests/conftest.py``. Shared steps can be referenced in a feature file without being imported into the corresponding ``test_`` file. Shared fixtures can be used in ``test_`` file functions by specifying the fixture name as an argument. Using a Tango client -------------------- The tests connect to the Tango devices using one of two mechanisms: - :external+pytango:doc:`Tango DeviceProxy ` - :external+ska-tango-tangogql:doc:`Tango GraphQL (TangoGQL) ` Each has been wrapped with the same methods (code defined in ``tests/common/tango``), and an environment variable controls which one is used in a specific environment. Set ``TEST_TANGO_CLIENT`` to ``dp`` to use DeviceProxy or ``gql`` for TangoGQL. Tests running in the CI pipeline (inside the same cluster as the SDP deployment) use the DeviceProxy client. Manual tests running outside the cluster against an SDP deployment in a local Minikube cluster or a remote cluster need to use the TangoGQL client. Values files for testing ------------------------ Two values files have been provided in\ ``resources/values`` to deploy the SDP for testing in different environments: - ``test-ci.yaml`` is used in the CI pipeline with the DeviceProxy client. - ``test-external.yaml`` is intended to be used with the TangoGQL client in Minikube or remote clusters. Running the tests ----------------- In the following, we will assume that you installed SDP into the environment of your choice, to the namespace of your choice, together with a Persistent Volume Claim for both the control system and the processing namespace (this you can achieve if you use the one of the values files described above as your custom values file when running the ``helm install`` command). The tests are marked with pytest markers. The ``TEST_MARKER`` environment variable specifies which tests will run when using the make targets. For example, if you only want to run the visibility receive test for the Low telescope, you would need to use ``TEST_MARKER="XTP-28695"``. Export the relevant namespace environment variables (update accordingly): .. code-block:: console $ export KUBE_NAMESPACE= $ export KUBE_NAMESPACE_SDP= Some of the tests described below are slightly different whether you are running SDP in minikube or in the DP Platform. We will clearly mark the differences. If you want to use the DP Platform, note that you will need VPN access. Please follow the instructions in `Confluence `__ (make sure you request access to the DP Platform - the TechOps cluster is a different VPN). When running the tests on the DP Platform, you have to be connected to the VPN. VPN is not needed for minikube. For the DP Platform, you also need KUBECONFIG access (assuming you installed SDP, you already have exported the file as needed.): .. code-block:: console $ export KUBECONFIG= Do not run this export for minikube. If you have exported a file, unset the variable: .. code-block:: console $ unset KUBECONFIG Set the ingress URL. For minikube: .. code-block:: console $ export TEST_INGRESS=http://$(shell minikube ip) For the DP Platform: .. code-block:: console $ export TEST_INGRESS=http://k8s.sdhp.skao .. note:: If you are using Minikube with the docker driver on a MacOS, then you must enable tunnelling for the tests to work, which is done by running this command in a separate terminal: .. code-block:: console $ minikube tunnel It may ask you for an administrator password to open privileged ports. The command must remain running for the tunnel to be active. You can run the tests with: .. code-block:: console $ make test .. note:: The visibility receive test requires a couple of helper pods, which connect to persistent volumes. These contain MeasurementSet data (stored using Git LFS in the repository, see :ref:`troubleshooting`), which are used for sending and validating the received data. These pods are automatically created and removed by the test. Testing in CI/CD ---------------- Jobs on every commit ^^^^^^^^^^^^^^^^^^^^ The SDP integration repository runs the tests automatically as part of its CI/CD pipeline in GitLab. See the `pipelines page `__. By default, on every commit, the test jobs are split into three parallel streams - on three separate subarray devices (``test-sdp/subarray/01``, ``02`` and ``03``) 1. Jobs tagged in their BDD feature files with ``SKA_mid`` in test job ``k8s-test-mid`` 2. Jobs tagged in their BDD feature files with ``SKA_low`` in test job ``k8s-test-low`` 3. All other jobs run in a ``common`` test job called ``k8s-test-common`` Alternatively, one can run all of the tests on a single subarray by adding the ``TEST_IN_SERIAL`` variable to the CI pipeline (can be done in GitLab or on a branch in code). There is a ``k8s-test-setup`` CI job which installs SDP in the ``KUBE_NAMESPACE`` and which runs in the stage ``pre-test`` and a corresponding ``k8s-test-cleanup`` which removes this deployment and runs in the ``post-test`` stage All of the tests, except the one marked as “extended_visibility_receive_test”, run on every commit. If new tests are being designed it is important to remember that concurrent test instances may be running on different subarrays. Existing tests incorporate their ``SUBARRAY_ID`` into any local named resources used for testing. Scheduled jobs ^^^^^^^^^^^^^^ Currently, two scheduled jobs are set up to run tests, which can be found here: `Pipeline Schedules `__ 1. Daily: Runs all of the tests (except “alternating_scan” ones) once a day, in the middle of the night (UK time). 2. Persistent SDP (DP Platform): Runs three times a day at 8, 14, 23 UTC, Monday to Friday. It only runs the extended (20-minute long) visibility receive test, a QA display test and the Mid pointing calibration test, in the ``dp-shared`` namespace on the DP Platform (see Section “Testing persistent deployments”). Testing persistent deployments ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ At the moment, there are two persistent deployments of SDP, running in two different namespaces on the `Data Processing Platform `__. dp-shared """"""""" The ``dp-shared`` (main) and ``dp-shared-p`` (for processing scripts) namespaces host the first persistent SDP deployment, which is used for manual testing, experimenting, and scheduled testing. It runs with two subarray devices, the first can be used for manual runs, while the second is used for scheduled and automated testing. The deployment is automatically updated after every merge to the master branch of ska-sdp-integration. Schedules run three times a day on the master branch (See `Persistent SDP (DP Platform) `__ schedule) and they execute vis-receive, QA displays, and pointing calibration related tests. sdp-staging """"""""""" The ``sdp-staging`` (main) and ``sdp-staging-p`` (for processing scripts) namespaces host the Staging deployment of SDP. This deployment is upgraded every time the SDP helm chart is released with a new tag. This allows for continuous deployment of new code into a running system.