BPP
===

The Batch Preprocessing Pipeline (BPP) applies a configurable chain of transformations to visibility
data read from storage, in order to prepare:

1. Calibrator observations for the instrumental calibration (INST) pipeline
2. Target observations for the self-calibration (ICAL) pipeline

Case 1: Preparing Calibrator observations
-----------------------------------------

.. mermaid::

    flowchart LR
        storedvis[("Stored Visibilities<br>(Calibrator)")] --> sf
        subgraph bpp["Batch Pre-Processing (BPP)"]
        sf["Static<br>flagging"] --> df["Dynamic<br>flagging"] --> bss["Bright source<br>subtraction"]
        end
        bss --> outvis[("Pre-Processed<br>Visibilities<br>(Calibrator)")]
        outvis --> inst(("Instrumental<br>Calibration<br>(INST)"))
        inst --> caltable[("Gain Table<br>(Calibrator)")]

BPP prepares calibrator observations for INST, which includes flagging bad visibilities and
optionally subtracting bright, out-of-field sources which would degrade the effectiveness of other
batch pipelines.

Case 2: Preparing Target observations
-------------------------------------

.. mermaid::

    flowchart LR
        storedvis[("Stored Visibilities<br>(Target)")] --> sf
        subgraph bpp["Batch Pre-Processing (BPP)"]
        sf["Static<br>flagging"] --> df["Dynamic<br>flagging"] --> bss["Bright source<br>subtraction"] --> ac["Applycal"] --> avg["Averaging"]
        end
        avg --> outvis[("Pre-Processed<br>Visibilities<br>(Target)")]
        outvis --> ical(("Self-Calibration<br>(ICAL)"))
        caltable[("Gain Table<br>(Calibrator)")] --> ac

BPP also prepares target field observations for ICAL to process. This additionally involves:

1. Applying to the target field visibilities the gains derived by INST on the most recent calibrator
   observation.
2. Averaging the visibilities in time and frequency to reduce the compute cost of ICAL.

Useful Links
------------

- **Repository**:
  https://gitlab.com/ska-telescope/sdp/science-pipeline-workflows/ska-sdp-batch-preprocess
- **Documentation**: https://developer.skao.int/projects/ska-sdp-batch-preprocess/en/latest/
- **Quickstart**: https://developer.skao.int/projects/ska-sdp-batch-preprocess/en/latest/aws.html
- **Status**: In development
- **Contact**: Team MERLIN
- **Where to get help**: `#merlin-batch-preprocessing
  <https://skao.slack.com/archives/C09HSJ2MH5K>`_