Changelog

3.0.0

New features

  1. Use ScienceDataModel for artefact management by default.

    1. Update run and run-from-sdp subcommands to accept SDM

    2. Initialise SDM stage to support run subcommand.

  2. Update target calibration workflow to also include rapthor-di calibration.

2.1.0

  1. Use default values for configuration having values defined by strategy for all INST pipelines.

  2. Update default values for some continuum imaging parameters.

2.0.2

  1. Fix piper dependency

2.0.1

  1. SKB-1312: Disable delay calibration stage in instrumental calibration pipeline

2.0.0

New features

  1. Added new steps inside "target_calibration" stage:

    1. complex_gain_calibration

    2. ionospheric_calibration

  2. Use Rapthor's new "slurm_static" mode when running e2e on multiple nodes.

  3. Use Rapthor for DD calibration as DI calibration will be done using complex_gain_calibration.

  4. Add Multi-scans input support for single source of calibrator and target observations.

  5. Support additional configuration for CIMG stage.

Breaking

  1. "max_selfcal_loops" parameters is now part of "self_calibration" nested configuration.

  2. Update Json config contract for run_from_sdp sub-command to match with pipeline stages.

  3. Update config contract for

    • run sub-command to pass multiple ms file paths for given calibrator source.

    • Additional configuration for CIMG stage.

General

  1. Update all stages as per the ska-sdp-exec-piper V2 API

  2. Disable resource monitoring in all of the sub-pipeline runs.

  3. Added support for INST v0.8+.

1.0.0

SKA SDP E2E CLI

  • Create a python-based CLI for running batch end-to-end pipelines. More details on the CLI are in RTD

Breaking

  • Batchlet and batchlet plugins modules to new repository at https://gitlab.com/ska-telescope/sdp/ska-sdp-exec-batchlet.git.

  • Remove streamlit sdp interface module.

0.4.2

Batchlet

  • Add a new option dask_report_dir which enables and stores dask performance reports. More details are in the batchlet documentation

0.4.1

Batchlet Plugins

  • Read hostname from logs, and include it in the benchmon csv

Batchlet

  • Remove monitoring results for failed pipeline runs.

0.4.0

Batchlet Plugins

  • Added batchlet filter plugin for SKA SDP standard logging format

  • Added consumer plugins for SKA SDP Configuration database (etcd) and simple csv file

  • The plugins are part of the ska_sdp_batchlet_plugins package. Detailed documentation is available here

Batchlet

  • Add support for log monitoring in batchlet

  • Add support for adding custom filter and consumer plugins, used for monitoring of the logs

  • (Breaking change) Update the monitor section of the input configuration schema. Please see the configuration details in the batchlet documentation

  • The DaskSlurmCluster class can take a decision appropriate parameters for the dask cluster based on allocated/available resources, similar to LocalCluster. Detailed documentation is available here

E2E Scripts

  • Added Rapthor as part of e2e scripts

  • The e2e.sh now calls other individual pipeline scripts as subprocesses.

  • Added suppport for completely spack based execution, so user does not have to clone the scripts from git. The steps to run the scripts are updated in the documentation

0.3.2

Batchlet

  • DaskSlurmCluster now inherits dask.distributed.SpecCluster class. This allows DaskSlurmCluster to be used in manner similar to other clusters in dask ecosystem (LocalCluster of SSHCluster). This also adds ability to gracefully close the cluster upon exit.

  • Add support for ska-sdp-benchmark-monitor in non-slurm (local) execution of batchlet.

E2E Scripts

  • Write output of instrumental calibration pipeline in the same directory as OUTPUT_DIR , instead of creating a unique timestamped subdirectory.

0.3.1

Batchlet

  • Force dask slurm cluster to start scheduler on random port, and use a unique scheduler file

  • Allow batchlet to start a local cluster when slurm environment is not available

  • Fix Typer's click dependency issue.

E2E scripts

  • Update slurm scripts to use spacks with metamodules

0.3.0

  • Add batchlet tool to run batch pipelines. The documentation for batchlet is available on readthedocs.

  • Removedbatch-continuum-imaging-pipeline cli tool, it is now replaced by batchlet.

  • The ska_sdp_e2e_batch_continuum_imaging package now contains slurm scripts, configuration and env setup files which are needed to run the end-to-end pipeline. Currently AWS environment is supported by default.

0.2.0

  • Removed dependency between pipelines and end-to-end batch continuum imaging pipeline.

  • End-to-End batch continuum imaging pipeline now functions as a tool which runs pipeline(s) as a single job. Dependency management delegated to the upstream process.

  • CLI Contract: End-to-End batch continuum imaging pipeline accepts a configuration file/stream and type of the configuration

  • Use srun for DASK cluster creation instead of dask-jobqueues in-order to use the resources allocated to the slurm job.

  • Add support for monitoring using ska-sdp-benchmark-monitor

0.1.0

Initial release of the SKA SDP end-to-end batch continuum imaging pipeline