Testing strategy

Introduction

ska-sdp-spack is a Spack package repository containing:

Spack packages (/packages)
Spack environments (/env and /os)
Python helper scripts (/scripts)

This page describes the testing strategy for each of these.

Python helper scripts

The scripts/ directory contains Python helper scripts. Tests for these scripts are in the tests/ directory. This repository uses the SKAO standard Makefile templates for running these tests, which means you can run these tests using:

make python-test

Duplicate package check

This repository duplicates some packages that are already in the upstream package repository from Spack itself. These packages overwrite the upstream package and add new versions and/or extra patches.

The scripts/check_upstream_duplicates.py script checks for Spack packages that are duplicates of packages in the upstream package repository from Spack itself. This check runs in CI to prevent accidentally adding duplicate packages.

If duplicating an upstream package is required, it must be added to the .ignore-duplicates file in the root of the repository. The scripts/check_upstream_duplicates.py script ignores the packages in that list.

Spack environments

The env/ and os/ directories contain Spack environments. Each environment defines a set of packages. Different tests run on CI for each environment:

os/
- Builds all packages in both Rocky Linux and Ubuntu Linux.
- Runs spack containerize to create a Dockerfile.
- Builds an OCI image using the standard SKAO CI templates.
flowchart LR A[Rocky Linux build] ~~~ B[Ubuntu Linux build] B ~~~ C[Create Dockerfile] C --> D[Build OCI image]

CI steps for the os/ environment
env/aws/ contains all packages that are deployed to the DP AWS cluster.
- Concretises the environment to check if concretising the packages in the enviroment is possible at all.
- Builds the concretised packages to check for build errors in the packages.
- Runs smoke tests for the built packages, which are described below.
flowchart LR F --> G subgraph Concretise job F[Concretise] end subgraph Build and test job G[Build] --> H[Run smoke tests] end

CI steps for the env/aws/ environment

Spack package smoke tests

As explained above, CI runs smoke tests for all packages in the env/aws environment. A smoke test is defined by adding a function with a test_ prefix, e.g. test_something to a spack package recipe.

Spack is very flexible regarding smoke tests and supports various types of smoke tests, see the Spack packaging guide. Each package can have multiple tests. The sections below describe the test types.

Python import tests

Classes that inherit from the PythonPackage class automatically inherit a test_imports test. This test searches for Python modules in the package and tests importing the found modules. For most Python packages, this automatic test is an adequate smoke test and does not require any changes to the package.

The test can be customized if necessary by adjusting the list of Python modules. The customizations below can be specified by defining the list directly or using a property function that generates the list. For example, import_modules = ["module"] defines a list directly. A property function can include or exclude modules depending on the package variant, for example. The py-py-key-value-aio package uses this approach.

Override all Python modules

A package can override the Python module list using import_modules. For example, the py-ducc package defines its own modules since Spack doesn’t detect Python modules that are binary wheels.

Exclude Python modules

A package can exclude modules from the Python module list using skip_modules. For example, the py-alembic package skips modules from an internal test suite. Importing such modules often fails because of extra requirements, like pytest, which are not needed during normal operation. The smoke test should therefore skip those modules.

Executable tests

Packages that install executables should test that those executables can run, for example by having the executable print a help message. A package can define such a test by adding a test_ function, for example:

    def test_application(self):
        """Ensure that 'application' runs."""
        application = which(self.prefix.bin.application)
        application("--version")

By default, the test succeeds when the application returns a 0 exit code. Testing an application with different input, output, or a custom exit code is also possible. The documentation for Executable::__call__ shows the various options.

Binary library tests

Packages that install binary libraries should test that the library can be used. If the package also installs executables that dynamically link to the library, running those executables also uses and thereby tests the library. Adding an executable test, as explained above, is then the easiest method of testing the library.

If an executable test does not suffice, a package can test the library by linking a small application to the library and running it. This works by creating an application in the test/ directory of the package, and adding a test that compiles and runs the test application. The test_register function in lofarstman package is a good example. It compiles and runs a simple C application.