Custom environments

Prerequisites

Make sure that your environment is set up to build software. This includes:

  • Git for getting sources

  • Working compilers for C, C++ and Fortran

  • Python 3.10 or later

For Ubuntu, the following should work:

$ sudo apt-get install -y \
    bash build-essential ca-certificates coreutils curl \
    file gfortran git gpg lsb-release python3 python3-pip \
    time unzip zip wget locales

The equivalent for Red Hat:

$ dnf install -y \
      automake bash bzip2 curl-minimal \
      diffutils file g++ gcc \
      gfortran git gpg libstdc++-static \
      patch python3 python3-pip unzip \
      wget xz zip glibc-langpack-en

Installation

Step 1: Install Spack

You can skip this step if you already have Spack installed, but make sure that you have the correct version, or the installation might not work:

$ git clone --depth=2 --branch=releases/v1.1 https://github.com/spack/spack.git
$ . spack/share/spack/setup-env.sh

You will have to repeat . spack/share/spack/setup-env.sh in the future if you want to run any further Spack commands.

Step 2: Clone ska-sdp-spack, create environment

Check out ska-sdp-spack and create a new environment:

$ git clone https://gitlab.com/ska-telescope/sdp/ska-sdp-spack.git
$ cd ska-sdp-spack
$ spack env create --without-view env/my_env
$ spack env activate env/my_env

If you want to start from an existing pre-built environment, you can also pass --include-concrete env/aws to spack env create. Proceed to configure the environment:

$ spack repo add .
$ spack buildcache add --unsigned gitlab-mirror oci://registry.gitlab.com/ska-telescope/sdp/ska-sdp-spack/spack-buildcache

Step 3: Add packages

At this point, add the packages to your environment that you would like to be installed. You can use spack add, or edit the env/my_env/spack.yaml file, for instance:

spack:
  specs:
  - matrix:
    - [ dp3+python@6.4.1, python@3.11 ]
    - [ "%gcc@11.4.0 target=x86_64_v4" ]
[...]

Check Spec Syntax in Spack documentation for details on how to specify packages to be installed. It is generally a good idea to be explicit about:

  • the versions of the software you want (check spack versions)

  • the compiler to use (Spack defaults to the newest available, but that might not be compatible with all packages, so it’s better to be conservative)

  • as well as architecture – you might want an appropriately optimised build, but still retain the possibility of using build caches.

Step 4: Concretise and install

Once you have declared all dependencies, you will need to allow Spack to detect your system compiler, and bootstrap the concretiser:

$ spack compiler find && spack bootstrap now

You will only need to do this once. Afterwards, you can concretise and install your environment:

$ spack concretise

Make sure to check the concretisation to ensure that you are getting the versions of packages you want. If you don’t – or concretisation fails – it is typically because of version conflicts between the involved packages. In those situations, it is generally best to simplify your environment until you can pin-point the conflicting packages.

However, once concretisation is done, you can proceed to installing your packages:

$ spack install

Step 5: Use software

At this point the software is installed into Spack directories. To use the software, load it into the environment as follows:

$ spack load wsclean
$ wsclean

This depends on Spack and the Spack environment being loaded. The following will work directly:

$ spack/bin/spack -e ska-sdp-spack/env/aws load wsclean
$ wsclean

Note that depending on the package you load, this might bring a number of software packages into scope. Especially be mindful that this might override your Python installation with a Spack-provided one!

Maintenance

To update to a newer version of ska-sdp-spack, pull new changes from Git and re-install changed packages:

$ cd ska-sdp-spack
$ git pull origin

At this point you might have to re-concretise your packages:

$ spack env activate env/my_env
$ spack concretise

Check again that the concretisation makes sense, see step 4 above. If you want to force Spack to re-concretise dependencies, you may need to pass --force and/or --fresh options to spack concretise. Then proceed to re-install your software:

$ spack install

You might also want to consider cleaning up temporary files and older installed packages to reclaim disk space:

$ spack clean
$ spack gc

Package Version Specification

To install specific versions of packages:

# Install specific WSClean and IDG versions
spack install wsclean@3.6 ^idg@0.8.1

# Check available versions
spack info wsclean

Troubleshooting

  1. Build Failures

    • Check system dependencies. System dependencies such as glibc version and/or CPU architecture may result in unsuccessful compilation.

    • Verify compiler compatibility. Some software may not be designed to be compiled with specific compiler, e.g. Intel OneAPI compiler suite.

    • Review build logs in ~/.spack/var/spack/builds/.

    • Build with debugging and verbose mode enabled: spack -d install -v.

  2. Version Conflicts

    • Use spack spec -I to inspect dependency resolution.

    • Consider using spack concretize --force for testing.

  3. If you encounter CUDA compatibility issues, you can specify an older CUDA version:

    # Install with specific CUDA version
spack install cuda@10.0.130
# Install idg with the specific version of CUDA
spack install idg ^cuda@10.0.130