Installation example using ska-sdp-spack

This page explains how to install the self-calibration pipeline. You probably have to adjust the instructions to your local environment. This page was originally copied from confluence. Later it was updated so it refers to the new ska-sdp-spack repository, which replaces schaap-spack.

Step 1: Install ska-sdp-spack

Clone the spack and ska-sdp-spack repositories.

cd ${HOME}

git clone --branch v0.23.0 https://github.com/spack/spack.git
source ./spack/share/spack/setup-env.sh

git clone https://gitlab.com/ska-telescope/sdp/ska-sdp-spack.git
spack repo add ./ska-sdp-spack

If spack is already available system-wide, make sure your custom installation is used instead of the system-wide version. For example, add the following line to ~/.bashrc:

source ${HOME}/spack/share/spack/setup-env.sh

Building schaap software requires the gcc compiler. Depending on your system, you may have to load a module for adding (a newer version of) the compiler to your PATH.

module load gcc/11

Now spack can build and install the necessary software. Use the latest DP3 and WSClean versions, since the pipeline may not work with older versions.

spack install dp3@latest wsclean@latest

Step 2: Install the pipeline

Ensure that correct Python executable is in your PATH.

spack load python
which python

Install poetry. There are many methods of installing poetry:

Using pipx (preferred method)

Optional: You may have to install pipx first, for example, using pip. (Installing pipx via spack is also possible, but we never tested it.)

python -m ensurepip # Install pip, if you don't have it.

# Ensure you're using the correct version of pip. It should match the Python version.
which pip
pip --version

# When 'pip' starts an incorrect version, use 'pip3.11' when e.g. using Python 3.11.x.
which pip3.11
pip3.11 --version

pip3.11 install pipx    # Install pipx, if you don't have it.

Then, install poetry:

pipx install poetry

Optional: If you installed pipx using pip, you can remove the installed packages.

pip3.11 list           # List installed packages via pip.
pip3.11 uninstall pipx # You may want to remove other installed packages as well.

Using pip
```
pip install poetry
```

Using spack (not tested, try at your own risk)

spack install py-poetry
spack load py-poetry

Clone the repository.

git clone --recursive git@gitlab.com:ska-telescope/sdp/science-pipeline-workflows/ska-sdp-wflow-selfcal.git

Create and activate a virtual environment.

cd ska-sdp-wflow-selfcal
poetry shell

Install the pipeline and its dependencies.

poetry install

Check your installation by running the pipeline without arguments, or by running one of the tests, for example:

ska-sdp-wflow-selfcal                  # Should print a usage message.
pytest tests/test_support_functions.py

Step 3: Run on a single node

Now that everything is installed, each time you want to run the pipeline you can follow these steps:

# Activate the virtual environment for the pipeline.
cd ${HOME}/ska-sdp-wflow-selfcal
poetry shell

# Unload all modules, since modules for Python packages override the
# packages installed via poetry.
module purge

Run, for example:

# Retrieve the installation directories for DP3, WSClean and EveryBeam.
EVERYBEAM_PREFIX=`spack find --format "{prefix}" everybeam`
DP3_PREFIX=`spack find --format "{prefix}" dp3`
WSCLEAN_PREFIX=`spack find --format "{prefix}" wsclean`

# Work around an OpenBLAS threading issue.
export OPENBLAS_NUM_THREADS=1

# Tell EveryBeam where the files with coefficients are.
export EVERYBEAM_DATADIR=$EVERYBEAM_PREFIX/share/everybeam

# Start the self-calibration pipeline.
# (Replace the various "/path/to" paths!)
ska-sdp-wflow-selfcal \
  --input_ms /path/to/midbands_averaged.ms \
  --work_dir /path/to/your/workdir \
  --dp3_path $DP3_PREFIX/bin/DP3 \
  --wsclean_cmd $WSCLEAN_PREFIX/bin/wsclean \
  --config /path/to/LOFAR.averaged.yml \
  --resume_from_operation calibrate_1 \
  --ignore_version_errors True

Possible errors you may encounter

No module named ‘numpy.core._multiarray_umath’ → solve with: export PYTHONPATH=””
ska-sdp-wflow-selfcal not found → Make sure you have installed the pipeline correctly.

When you are in the poetry shell, pip list should show a line like:
```
ska-sdp-wflow-selfcal              0.2.0       /path/to/ska-sdp-wflow-selfcal
```
You can also try running the pipeline directly, by explicitly specifying the path to python and the path to the pipeline’s main.py file. This way, you’ll also ensure you’re using the correct python executable.
```
/path/to/python src/ska-sdp-wflow-selfcal/main.py \
--input_ms /path/to/midbands_averaged.ms \
--work_dir /path/to/your/workdir \
--config /path/to/LOFAR.averaged.yml \
--resume_from_operation calibrate_3
```
Segmentation fault: A common cause is mixing different versions of the CasaCore or EveryBeam library.
- The pipeline will detect this situation and print errors.
- The –ignore_version_errors argument ignores these errors. In practice, this argument is always necessary, since using the same versions everywhere is very hard.
- The pipeline uses these C++ libraries directly, using their Python bindings.
- The EveryBeam library uses the CasaCore library. From version 0.6.1, the EveryBeam Python library uses a custom CasaCore library where all CasaCore functions use a different C++ namespace, which avoids conflicts with other CasaCore libraries.
- Both DP3 and WSClean use both Casacore and EveryBeam. Since the pipeline runs DP3 and WSClean in different processes, using different library versions in these applications shouldn’t be a problem.