SDM Mode Guide

This section is intended for running the Batch Pre-processing Pipeline (BPP) within the SDP production system. It describes how to run BPP in SDM mode, where the pipeline exchanges data with other pipelines via a shared Science Data Model (SDM) directory.

If you are running BPP as a standalone tool — reading Measurement Sets from disk and writing outputs to an output directory of your choice — see the Quickstart and Configuration Guide instead.

Note

The Demixer step is not available in SDM Mode yet.

Overview

In SDM mode, BPP:

• Reads calibration tables and other step inputs from the SDM directory.

• Writes logs, QA products and the run configuration into the SDM directory, under a uniquely-indexed subdirectory of <sdm>/logs/.

• Writes processed Measurement Sets to the --output-dir as usual.

CLI Invocation

Pass --sdm-path for SDM mode:

ska-sdp-batch-preprocess run \
    --config myConfig.yaml \
    --output-dir /path/to/output_dir \
    --sdm-path /path/to/sdm_directory \
    input1.ms input2.ms ...

Output Directory Conventions

When --sdm-path is provided, a new indexed subdirectory is created under <sdm>/logs/:

<sdm>/
└── logs/
    └── NN-bpp/   # NN = next available index (01, 02, …)
        ├── pipeline.log
        ├── config.yaml
        ├── task-list.json
        ├── dask-report.html
        └── qa/
            ├── <input_stem>_flagging_report.zarr
            └── <input_stem>_flagging_report.png

Processed Measurement Sets are written to --output-dir as in standalone mode.

Configuring Steps in SDM Mode

Applycal

In SDM mode, specify a table block with kind: sdm, field_id and purpose. BPP resolves the calibration table at:

<sdm>/calibration/<purpose>/<field_id>/gaintable.h5parm

Example configuration:

steps:
  - step: applycal
    table:
      kind: sdm
      field_id: target_field
      purpose: bandpass

AOFlagger

Strategy presets should be used when using the AOFlagger step in SDM mode. See Pipeline Steps and AOFlagger Strategy Presets for details.

Demixer

Note

Sky model lookup from the SDM is not yet implemented. Further adjustments to the SDM and sky model schemas must be discussed before SDM-mode Demixer is implemented.