Developing Spack packages

Prerequisites

This page assumes you have a working Spack installation and you can for example run spack install dp3. The main README describes setting up Spack.

Git workflow

The Git repository is https://gitlab.com/ska-telescope/sdp/ska-sdp-spack/ . To propose changes to ska-sdp-spack, the general workflow is:

  1. Create a feature branch

  2. Submit a merge request

  3. Assign reviewer(s):

    • The CODEOWNERS file describes who is responsible for the part of the code.

    • Ask in the #ska-sdp-spack slack channel.

  4. If your changes alter spack’s dependency resolution the CI/CD pipeline will fail. Check the concretization makes sense before downloading and commiting the new spack.lock file and ensure the pipeline passes.

    For example:

    $ git clone git@gitlab.com:ska-telescope/sdp/ska-sdp-spack.git
$ git checkout -b team-123-make-change
... make changes ...
$ git push origin
... will display a link to create merge request ...

Packages and environments

The ska-sdp-spack repository maintains both a Spack package repository as well as a number of Spack environments.

  • Environments combine ready-made collections of package (versions) and configurations that can be installed directly (see Pre-built environments). This will especially be used for building software in Continuous Integration, and is the suggested starting point for users that don’t want to build their own environments. See

  • Packages provide abstract recipes to build software required for the SKA SDP, including dependencies. Using Spack’s dependency solver, package definitions alone should typically be enough for users to be able to generate working software environments (typically by compiling the software themselves).

They are represented as follows in the repository::

ska-sdp-spack/
  ├── env/
  |   ├── aws/
  │   │   ├── common.yaml
  │   │   ├── definitions.yaml
  │   │   ├── packages.yaml
  │   │   ├── rhel9/
  │   │   │   ├── spack.yaml
  │   │   │   └── spack.lock
  │   │   ├── ubuntu/
  │   │   │   ├── spack.yaml
  │   │   │   └── spack.lock
  │   │   └── ...
  └── packages/
      ├── py-ska-sdp-config/
      │   └── package.py
      ├── py-ska-sdp-datamodels/
      │   └── package.py
      ├── py-ska-sdp-func/
      │   └── package.py
      └── ...

The YAML files in env/aws contain definitions that are common across the different environment flavours:

  • common.yaml: Common settings for concretiser and repositories (especially which Spack package repository version to use)

  • definitions.yaml: List of software to include. There are three categories:

    • cmake_software: CMake packages

    • python_software: Python packages

    • compiled_software: Compiled packages (not using CMake)

  • packages.yaml contains constraints on important common dependencies, such as mathematical and astronomy utility libraries.

Registering environments to CI/CD

If you add new environments in env/aws/ and you want ci/cd to consider them, you need to add the new environment to the matrix in .gitlab-ci.yml.

For example:

.spack-env-matrix: &spack-env-matrix
  parallel:
    matrix:
      - ... other ...
      - DISTRO: rhel9
        SPACK_ENV_PATH: env/aws/rhel9-python3.13

Similarly, if you add new distros in images/, you need to add the new distro to the matrix in .gitlab-ci.yml.

For example:

.distro-matrix: &distro-matrix
  parallel:
    matrix:
      - DISTRO: [..., new-distro]

Package conventions

  • See also the general conventions for this repository.

  • ska-sdp-spack uses the standard SKAO Python code formatting and linting tools. Since Spack packages are written in Python, all ska-sdp-spack packages have to pass these checks.

  • When creating a package version that points to a branch, use the branch name for the version name. I.e. if you support compiling from main, call the version main (not e.g. develop).

  • When creating a package version from a Git commit, tag the base version with the commit date in YYYYMMDD format. For example, the wsclean package defines version 3.6.20260109. Using such tags ensures corect version ordering, e.g., 1.2.3 < 1.2.3.20250320 < 1.2.3.20251003 < 1.2.4.

Creating packages

New packages

Creating a new package works as follows:

  1. Create a new directory for the package in the packages/ directory. Note that Python package names start with a py- prefix, e.g., py-numpy.

  2. Add the package directory to the CODEOWNERS file.

  3. Add a package.py file to the new directory. The easiest method of creating a package.py file is copying an existing package.py file and adjusting it:

    • The class name needs to follow your package’s name.

    • Update the homepage, git, maintainers, and license. If it’s availalbe on pypi, also update the pypi line as well.

    • Add the version(s) you want to make available. You don’t need to add every existing version. Just add the one(s) most likely to be needed.

    • If you are building the software from a Git repository, make sure to add explicit commit hashes for each version so concretisation doesn’t change when the Git repository isn’t available.

    • Update the depends_on lines, which describe the dependencies of your package.

      • Spack has its own syntax for specifying version ranges. For example, @2.3:2 means the minimum version is 2.3.0 and the maximum major version is 2. Versions 3.x are thus not allowed. Follow these instructions for setting versions in dependencies.

      • For Python packages the pyproject.toml file typically describes the dependencies. The Spack package should then specify the same dependencies for the version or commit you are using.

    • The Spack packaging guide describes more details on how to create packages.

  4. Add the package to one or more environments or make it a dependency of an existing package. When skipping this step, CI will not build the package (see above).

Overriding packages

ska-sdp-spack contains several packages that override packages in the upstream spack-packages repository. There are various reasons for overriding packages:

  • You need a newer version than exists in spack-packages

  • You need a new variant that doesn’t exist in spack-packages, or need to adjust the version constraints or dependencies.

  • You need to apply an extra patch.

  • The package in question does exist in the develop branch of spack-packages, but hasn’t been released yet.

When overriding a package:

  1. Copy the upstream package directory, including all content, to the packages/ directory.

    • You should replace underscores in the directory name by hyphens, e.g., py_key_value_aio becomes py-key-value-aio.

  2. Make the required changes

  3. Open a pull request in spack-packages to upstream the change so that we do not have to maintain it ourselves in ska-sdp-spack.

    • Consider becoming the maintainer of the package in spack-packages.

  4. Add the package directory to the CODEOWNERS file.

  5. Update the package.py file:

    • Clearly state in comments why the override exists, and when it should be removed (e.g. provide a link to the upstream pull request).

    • Make the package compatible with the formatting and linting tools.

      • Making the package compliant typically requires only few changes:

        • Adjust the from spack.packages import * line.

        • Run the Python formatter, e.g., using make python-format.

    • Update the package itself. For example, add versions, patches, and/or dependencies. Add a comment near each change, so it’s clear where the package differs from the original package.

  6. Add the package to the .ignore-duplicates file. See the testing strategy.

Non-release versions

You may run into a situation where you need to make a version of the software available through ska-sdp-spack that is not (yet) available as an official release. For instance, we might depend on a bug fix to be released, or a new experimental feature to become available. In those cases, consider the following courses of action (in this rough order):

  1. Enquire with the repository maintainers whether a release could be made. Especially with SKA-affiliated organisations, there is a good chance that we might have somebody in the organisation that has sufficient rights. When in doubt check the CODEOWNERS file, and raise the concern with the program team if you get stuck.

  2. If a release cannot be made, add a “pseudo”-release to Spack pointing at a specific commit in the repository in question. The new version should include a date tag, as described in the conventions.

  3. If the change in question is not in the repository and there is good reason to believe that the change would never be included in the repository (e.g. because it works around a problem that is specific to SKAO), consider applying patch files or using a patch() function. A patch even allows changing an existing release.

    • Applying a patch file works as follows:

      • Create a .patch file, e.g. using diff -Naur <original file> <patched file>

      • Store the .patch file in the same directory as the package.py file for the package.

      • Add a patch() call to the package.py file. For example, patch("filename.patch", when="@:0.42.0") applies filename.patch on all versions up to and including version 0.42.0.

    • A patch() function in package.py can contain arbitrary Python code. Spack provides the filter_file() function for applying patches. See the Spack documentation.

Installing new packages

After creating a new package (version) or an override package, follow these steps:

  1. Add your package to, or bump the version in:

    • definitions.yaml, if it is an application

    • packages.yaml, if it is a common dependency that we want to have a specific version. If we do not care about the version, we do not need to specify it as Spack will automatically pull it in when it sees a package.py that references it.

  2. (Optional) Test installing the package locally, for example:

    • Only install the new package: spack install <package>

    • (Re)concretise an environment containing the package:

      spack env activate <environment name/path>
spack concretise -f
spack install

  3. Create a merge request. See the Git workflow. CI will run a pipeline.

  4. If you updated a package in the env/aws environment or this environment itself, the CI pipeline probably fails at the first run. You’ll have the update the spack.lock file for this environment. Proceed as follows:

    1. Wait until the concretise-aws-[rhel9,ubuntu] jobs finish, which takes a few minutes. Waiting for the entire CI pipeline is not needed.

    2. The concretise-aws-[rhel9,ubuntu] CI jobs re-concretise the env/aws environment and generates a new spack.lock file per environment in env/aws. View the output of this job.

    3. Check that the new concretisation makes sense, by looking at:

      • The list of concretised packages.

      • The changes compared to the old concretisation.

    4. At the end of the job, it shows a link to the new spack.lock file. Download it, replace the existing env/aws/spack.lock file by it.

    5. git commit the new spack.lock file and git push the change.

    6. The CI pipeline should succeed now.

Troubleshooting

This list describes a few problems we encountered and how to fix them. It is not exhaustive.

Changed spack.lock file.

The build-aws CI job may fail with the following message:

=================================================================================
The env/aws/spack.lock file changed after 'spack install', which means that it
did not implement all specs from spack.yaml. Check whether the concretisation above
makes sense, and if so commit the spack.lock generated from this CI job:
https://gitlab.com/ska-telescope/sdp/ska-sdp-spack/-/jobs/10590011105/artifacts/raw/env/aws/spack.lock
=================================================================================

If this happens, update the env/aws/*/spack.lock files by following the procedure described in installing new packages.

Errors in build-aws

Other errors may occur in this CI job. Some suggestions:

  • Search for error and see what it comes up with.

  • Usually the outputs are saved in txt files, which are also available in the job output. Inspect the file to see what needs fixing.

  • Try reproducing the issue locally by installing the failed package and/or environment locally. See installing new packages.

Missing dependencies

You may get errors like Cannot satisfy 'py-jsonschema@4.23:' during concretisation. Note that concretisation also happens when installing a single package, without using an environment.

For solving this issue, spack versions <package_name> gets the versions that are available (“Safe versions (already checksummed)”) or not available (“Remote versions (not yet checksummed)”). Double check if the version is available on Spack or in the local repository you have.

  • If the version is available in those places, make sure you have added your local repo to spack and you ran spack concretise, too.

  • If the version is not available, you will need to add it.

    • If the package does not exist in ska-sdp-spack, create a new package, as described above.

    • If the package exists, update it by adding the required version. Also, update the package dependencies if needed. Different versions often have different dependencies.

Finding sha256 hashes

Non-SKAO packages versions are usually referred to by their sha256 hash. The build-aws job fails if this hash is incorrect. Example error message:

==> Error: ChecksumError: sha256 checksum failed for /tmp/root/spack-stage/spack-stage-py-python-benedict-0.34.1-dnbxazv4q3znilvtsgoelou3up4azkud/python_benedict-0.34.1.tar.gz
    Expected f09711ca2d8e716ca40440f1e26dfa3ccff15e08f1baa88de594fbba799cf87a but got 0a81784c826c8983c485bb647ed5e6c8fad139c3921ea1844b21e41da0ff0dc0

There are various means of getting the correct sha256 hash:

  • Copy the hash from the error message above.

  • For Python packages, get the hash value from PyPI:

    1. Find the package on PyPI.

    2. Go to Release history, and choose the version.

    3. Go to Download files,

    4. Use the view details link next to filename.

    5. Use the Copy button next to the SHA256 hash.