E2E
===

The SKA SDP Batch E2E Pipeline is a command-line interface designed to automate and run the batch
SDP pipelines on SKA HPC Cluster. It uses a YAML-driven configuration to orchestrate sub-pipelines,
managing execution via Slurm or standard shell environments.

The `CLI
<https://developer.skao.int/projects/ska-sdp-e2e-batch-continuum-imaging/en/latest/e2e_cli.html>`_
provides two primary entry points: the ``run`` subcommand for manual *HPC* operations and the
``run-from-sdp`` subcommand for automated processing scripts.

Inputs and Outputs
------------------

======== ===========================================================================================
Inputs:  Stored visibilities (MSv2), sky models (OSKAR CSV), and a configuration file defining stage
         parameters (YAML).
Outputs: Calibrated visibilities (MSv2) and science-ready continuum images (FITS).
======== ===========================================================================================

High-Level Stages
-----------------

The workflow follows a modular sequence to transform raw data into images:

======================== ==========================================================================
Instrumental Calibration Performs initial preprocessing and solves for instrumental gains using
                         calibrator visibilities. This stage runs :doc:`bpp` and :doc:`inst` as
                         sub-pipelines.
Target Calibration       Executes target-specific preprocessing and iterative self-calibration to
                         refine data quality. This stage runs :doc:`bpp`, Target :doc:`inst` and
                         :doc:`ical` as sub-pipelines.
Continuum Imaging        Synthesizes the final data into continuum images. This stage runs
                         :doc:`cimg` as a sub-pipeline.
Spectral Line Imaging    *(Yet to be integrated)* Runs continuum subtraction on corrected
                         visibilities and synthesizes spectral cubes. This will execute :doc:`simg`
                         pipeline.
======================== ==========================================================================

.. mermaid::

    flowchart LR
      calibrators[("Stored Visibilities <br> (Calibrator)")] --> e2e
      targets[("Stored Visibilities <br> (Target)")] --> e2e
      sky-model[("Sky Model")] --> e2e
      config[("Configuration <br> YAML")] --> e2e
      e2e --> contimg[("Continuum <br> Image")]
      e2e --> cor_vis[("Calibrated <br> Visibilities <br> (DI)")]
      e2e --> speccube[("Spectral Cube")]
      subgraph e2e["SKA SDP Batch E2E Pipeline"]

      inst(("Instrumental <br> Calibration")) --> gain-table[("Gain Table <br> (Calibrator)")] --> target-cal(("Target <br> Calibration"))
      target-cal --> facet_region[("Facet <br> Regions")] & field_sol[("Field <br> Solutions")] --> cimg
      target-cal --> corrected-vis[("Calibrated <br> Visibilities <br> (DI)")] --> cimg


      cimg(("Continuum <br> Imaging")) --> continuum-image[("Continuum <br> Image")] & model[("Model <br> Image")]

      model1[("Model <br> Image")] --> simg
      corrected-vis2[("Calibrated <br> Visibilities <br> (DI)")] --> simg
      simg(("Spectral Line <br> Imaging")) --> spec-cube[("Spectral Cube")]
      end

Useful Links
------------

- **Repository**:
  https://gitlab.com/ska-telescope/sdp/science-pipeline-workflows/ska-sdp-e2e-batch-continuum-imaging
- **Documentation**:
  https://developer.skao.int/projects/ska-sdp-e2e-batch-continuum-imaging/en/latest
- **Quickstart**:
  https://developer.skao.int/projects/ska-sdp-e2e-batch-continuum-imaging/en/latest/batch_scripts.html
- **Status**: In development
- **Contact**: Team DHRUVA
- **Where to get help**: `#help-sdp-batch-pipelines <https://skao.slack.com/archives/C0973F4MRRN>`_