Quickstart

This is a quick start guide to get you up and running with the CIMG pipeline on a local machine. We have provided a Dockerfile to make this as easy as possible. You will need to have Docker or podman installed on your machine.

Installation

  1. Open a terminal and navigate to the directory you want to clone the project.

  2. Clone the project repository using git.

    git clone --recurse-submodules https://gitlab.com/ska-telescope/sdp/ska-sdp-cimg.git
cd ska-sdp-cimg

  3. Start a podman (or docker) machine if you don’t have one already. Ensure it has at least 12 GB of RAM allocated (required for compiling casacore).

    podman machine init
podman machine set --memory 12000
podman machine start

Running the pipeline

The pipeline can be run using podman or the provided development container.

Using the development container

If you use VS Code, you can use the Dev Containers extension to automatically build and run the development environment. Simply open the project in VS Code and select “Reopen in Container” when prompted.

  1. Wait for the container to finish building (it might take around 30 minutes first time)

  2. Once the container is running, open a terminal in VS Code and run the following command to start the Prefect server and run the continuum imaging pipeline on a test dataset. The test dataset is small and should run in less than 10 minutes. The output will be stored in the current working directory in the output folder.

./scripts/test/start.sh

3. If you want to run the pipeline on your own data, you can modify the start.sh. You can provide your own paths to the configuration file, measurement set, solutions file, and facet regions file.

 poetry run python src/ska_sdp_cimg \
--config-path "config/cimg_config.yaml" \
--ms-path "tests/data/example.ms" \
--solutions-path "tests/data/example.solutions.h5" \
--facet-regions-path "tests/data/example.facet.reg" \
--tmp-dir "tmp" \
--output-dir "output"

If you don’t want to use the Prefect server, you can run the pipeline directly using the command above. Just paste the command with the appropriate paths in the terminal and it will run the pipeline using a temporary Prefect server.

Building the container image yourself

The provided Dockerfile can be used to build a container image using podman (or docker). The dev target can be built and used in the same way as the dev container (see above). For a simple test, there is also a test target which will start the prefect server and run the pipeline automatically on the test data and configs provided in the repository.

  1. Build the container image using the provided Dockerfile. This requires approximately 12 Gb of free disc space. The following will build the test target. You can replace test with dev if you want to experiment with or develop the pipeline (or to use the dev container in VSCode, see above)

podman build --target test -f Dockerfile -t cimg

  1. Run the container, mounting the current directory to /src in the container. Ensure that port 4200 is available on your machine. This port is used for the Prefect UI.

podman run -it -p 4200:4200 -v $(pwd):/app localhost/cimg

  1. The Prefect server will start automatically in the container. Wait for the Prefect UI to be available at http://localhost:4200. The code in main.py will run automatically when the container starts. You can monitor the progress of the flow in the Prefect UI.