INST
====

The Instrumental Calibration Pipeline (INST) is a CLI-based application designed to perform
calibration on SKA visibility data. It provides automated workflows for both instrumental and target
calibration, generating the initial products required for standard SKA batch processing.

The pipeline is highly flexible, supporting experimental stage reordering via YAML and deployment
through Docker, Spack, or standard Python environments. This pipeline relies on ``dask`` for
distribution of work across multiple processes or multiple HPC nodes.

The Instrumental Calibration Pipeline (INST) provides three distinct CLIs, each tailored to execute
specific calibration processes for calibrator and target visibility data.

**Inputs and Outputs**

======== ======================================================================================
Inputs:  Visibilities (MSv2), configuration files (YAML) and sky model (SKA LSM csv or GLEAM
         extragalactic).
Outputs: Calibration gain tables (h5parm), Zarr-formatted cache files, and diagnostic QA plots.
======== ======================================================================================

**High-Level Stages**

====================== ===========================================================================
Load and Prepare Data: Ingests MSv2 measurement sets and converts them into Zarr format, utilizing
                       Dask for parallelized resource management.
Visibility Prediction: Generates model visibilities from the input sky model.
Calibration:           Performs different calibration operations through dedicated solvers.
====================== ===========================================================================

.. mermaid::

    flowchart TD
      bpp(("Batch Pre-processing <br> Pipeline")) --> measurementset[("Pre-Processed<br>Visibilities<br>")] --> inst
      skymodel[("Global Sky <br> Model")] --> inst

      subgraph inst ["Instrumental Calibration"]
        direction LR
        load-data["Load and <br> Prepare Data"] --> predict-visibility
        predict-visibility["Visibility Prediction"] --> Calibration
      end

      inst --> gain-table[("Gain Table<br>")] --> bpp1(("Batch Pre-processing <br> Pipeline"))
      storedvis[("Stored Visibilities<br>")] --> bpp1

Instrumental Calibration pipeline
---------------------------------

==================== ======================================
Input Visibilities:  Pre-processed Calibrator visibilities.
Type of Calibration: Bandpass and delay correction.
==================== ======================================

Target Calibration pipeline
---------------------------

==================== ==================================
Input Visibilities:  Pre-processed Target visibilities.
Type of Calibration: Complex gains.
==================== ==================================

Target Ionospheric Calibration pipeline
---------------------------------------

==================== ==================================
Input Visibilities:  Pre-processed Target visibilities.
Type of Calibration: Ionospheric corrections.
==================== ==================================

Key Dependencies
----------------

The pipeline integrates high-performance external libraries for core operations:

================================================================ =================================
Library                                                          Description
================================================================ =================================
`everybeam <https://git.astron.nl/RD/EveryBeam>`_                Used for computing beam responses
                                                                 required for the predict stage.
`python-casacore <https://casacore.github.io/python-casacore/>`_ Provides python bindings for
                                                                 ``casacore`` library, and is
                                                                 primarily used for MSv2 table
                                                                 operations.
================================================================ =================================

Useful Links
------------

- **Repository**:
  https://gitlab.com/ska-telescope/sdp/science-pipeline-workflows/ska-sdp-instrumental-calibration
- **Documentation**: https://developer.skao.int/projects/ska-sdp-instrumental-calibration/en/latest/
- **Quickstart**:
  https://developer.skao.int/projects/ska-sdp-instrumental-calibration/en/latest/README.html#installing-the-pipeline
- **Status**: In development
- **Contact**: Team DHRUVA
- **Where to get help**: `#team-dhruva <https://skao.slack.com/archives/C071S0XF3RT>`_ ,
  `#help-sdp-batch-pipelines <https://skao.slack.com/archives/C0973F4MRRN>`_