SDM Mode

Overview

The INST pipeline supports Science Data Model(SDM), which integrates with the SDP by exchanging data with other pipelines via a shared SDM directory.

When --sdm-path is provided, the pipeline:

Writes gaintables into the SDM directory under <sdm-path>/calibration/<purpose>/<field_id>/ instead of the user provided output directory.
Writes logs and QA products under a uniquely-indexed subdirectory <sdm-path>/logs/NN-inst/ (where NN is the next available index).
Resolves the sky model from the SDM directory using the field ID extracted from the input Measurement Set. The file <sdm-path>/sky/<field_id>/sky_model.csv must exist.

If you are running the pipeline as a standalone and do not pass --sdm-path, pipeline works as usual, writing outputs and QA products in directory provided with --output.

Gaintables will be written at given location for each pipeline:

Instrumental calibration: bandpass/<field_id>/gaintable.h5parm
Instrumental target calibration: gains/<field_id>/gaintable.h5parm
Instrumental target ionospheric calibration: ionosphere/<field_id>/gaintable.h5parm

Note

When --sdm-path and --output are used together, but --sdm-path takes precedence for gaintable, sky model and log output locations. When omitted, the pipeline falls back to writing outputs under --output.

CLI Invocation

User can pass --sdm-path as follows while running any of the INST pipeline:

ska-sdp-instrumental-calibration run \
    --config config.yaml \
    --output /path/to/output_dir \
    --sdm-path /path/to/sdm_directory \
    input.ms

It's similar for the target calibration and ionospheric calibration pipelines.

Directory Conventions

When --sdm-path is provided, pipeline assumes following folder structure to be present in SDM directory

<sdm>/
├── sky/
│   └── <field_id>/
│       └── sky_model.csv
├── calibration/
├── logs/
└─ ...

Post-run, the pipeline updates the SDM directory like this:

<sdm>/
├── calibration/
│   └── <purpose>/
│       └── <field_id>/
│           └── <filename>.h5parm     # exported gaintable
└── logs/
    └── NN-inst/                      # NN = next available index (01, 02, …)
        ├── pipeline.log
        ├── config.yaml
        └── qa/