E2E

The SKA SDP Batch E2E Pipeline is a command-line interface designed to automate and run the batch SDP pipelines on SKA HPC Cluster. It uses a YAML-driven configuration to orchestrate sub-pipelines, managing execution via Slurm or standard shell environments.

The CLI provides two primary entry points: the run subcommand for manual HPC operations and the run-from-sdp subcommand for automated processing scripts.

Inputs and Outputs

Inputs:

Stored visibilities (MSv2), sky models (OSKAR CSV), and a configuration file defining stage parameters (YAML).

Outputs:

Calibrated visibilities (MSv2) and science-ready continuum images (FITS).

High-Level Stages

The workflow follows a modular sequence to transform raw data into images:

Instrumental Calibration

Performs initial preprocessing and solves for instrumental gains using calibrator visibilities. This stage runs BPP and INST as sub-pipelines.

Target Calibration

Executes target-specific preprocessing and iterative self-calibration to refine data quality. This stage runs BPP, Target INST and ICAL as sub-pipelines.

Continuum Imaging

Synthesizes the final data into continuum images. This stage runs CIMG as a sub-pipeline.

Spectral Line Imaging

(Yet to be integrated) Runs continuum subtraction on corrected visibilities and synthesizes spectral cubes. This will execute SIMG pipeline.

 
        flowchart LR
  calibrators[("Stored Visibilities <br> (Calibrator)")] --> e2e
  targets[("Stored Visibilities <br> (Target)")] --> e2e
  sky-model[("Sky Model")] --> e2e
  config[("Configuration <br> YAML")] --> e2e
  e2e --> contimg[("Continuum <br> Image")]
  e2e --> cor_vis[("Calibrated <br> Visibilities <br> (DI)")]
  e2e --> speccube[("Spectral Cube")]
  subgraph e2e["SKA SDP Batch E2E Pipeline"]

  inst(("Instrumental <br> Calibration")) --> gain-table[("Gain Table <br> (Calibrator)")] --> target-cal(("Target <br> Calibration"))
  target-cal --> facet_region[("Facet <br> Regions")] & field_sol[("Field <br> Solutions")] --> cimg
  target-cal --> corrected-vis[("Calibrated <br> Visibilities <br> (DI)")] --> cimg


  cimg(("Continuum <br> Imaging")) --> continuum-image[("Continuum <br> Image")] & model[("Model <br> Image")]

  model1[("Model <br> Image")] --> simg
  corrected-vis2[("Calibrated <br> Visibilities <br> (DI)")] --> simg
  simg(("Spectral Line <br> Imaging")) --> spec-cube[("Spectral Cube")]
  end