INST

The Instrumental Calibration Pipeline (INST) is a CLI-based application designed to perform calibration on SKA visibility data. It provides automated workflows for both instrumental and target calibration, generating the initial products required for standard SKA batch processing.

The pipeline is highly flexible, supporting experimental stage reordering via YAML and deployment through Docker, Spack, or standard Python environments. This pipeline relies on dask for distribution of work across multiple processes or multiple HPC nodes.

The Instrumental Calibration Pipeline (INST) provides three distinct CLIs, each tailored to execute specific calibration processes for calibrator and target visibility data.

Inputs and Outputs

Inputs:

Visibilities (MSv2), configuration files (YAML) and sky model (SKA LSM csv or GLEAM extragalactic).

Outputs:

Calibration gain tables (h5parm), Zarr-formatted cache files, and diagnostic QA plots.

High-Level Stages

Load and Prepare Data:

Ingests MSv2 measurement sets and converts them into Zarr format, utilizing Dask for parallelized resource management.

Visibility Prediction:

Generates model visibilities from the input sky model.

Calibration:

Performs different calibration operations through dedicated solvers.

 
        flowchart TD
  bpp(("Batch Pre-processing <br> Pipeline")) --> measurementset[("Pre-Processed<br>Visibilities<br>")] --> inst
  skymodel[("Global Sky <br> Model")] --> inst

  subgraph inst ["Instrumental Calibration"]
    direction LR
    load-data["Load and <br> Prepare Data"] --> predict-visibility
    predict-visibility["Visibility Prediction"] --> Calibration
  end

  inst --> gain-table[("Gain Table<br>")] --> bpp1(("Batch Pre-processing <br> Pipeline"))
  storedvis[("Stored Visibilities<br>")] --> bpp1

Instrumental Calibration pipeline

Input Visibilities:

Pre-processed Calibrator visibilities.

Type of Calibration:

Bandpass and delay correction.

Target Calibration pipeline

Input Visibilities:

Pre-processed Target visibilities.

Type of Calibration:

Complex gains.

Target Ionospheric Calibration pipeline

Input Visibilities:

Pre-processed Target visibilities.

Type of Calibration:

Ionospheric corrections.

Key Dependencies

The pipeline integrates high-performance external libraries for core operations:

Library

Description

everybeam

Used for computing beam responses required for the predict stage.

python-casacore

Provides python bindings for casacore library, and is primarily used for MSv2 table operations.