# Changelog

## 3.0.0

### New features

1. Use ScienceDataModel for artefact management by default.
   1. Update run and run-from-sdp subcommands to accept SDM
   2. Initialise SDM stage to support run subcommand.
2. Update target calibration workflow to also include rapthor-di calibration.

## 2.1.0
1. Use default values for configuration having values defined by strategy for all INST pipelines.
2. Update default values for some continuum imaging parameters.

## 2.0.2
1. Fix piper dependency

	
## 2.0.1
1. SKB-1312: Disable delay calibration stage in instrumental calibration pipeline

## 2.0.0

### New features

1. Added new steps inside "target_calibration" stage:
   1. complex_gain_calibration
   2. ionospheric_calibration
2. Use Rapthor's new "slurm_static" mode when running e2e on multiple nodes.
3. Use Rapthor for DD calibration as DI calibration will be done using complex_gain_calibration.
4. Add Multi-scans input support for single source of calibrator and target observations.
5. Support additional configuration for CIMG stage.

### Breaking

1. "max_selfcal_loops" parameters is now part of "self_calibration" nested configuration.
2. Update Json config contract for `run_from_sdp` sub-command to match with pipeline stages.
3. Update config contract for
   - `run` sub-command to pass multiple ms file paths for given calibrator source.
   - Additional configuration for CIMG stage.

### General

1. Update all stages as per the ska-sdp-exec-piper V2 API
2. Disable resource monitoring in all of the sub-pipeline runs.
3. Added support for INST v0.8+.

## 1.0.0

### SKA SDP E2E CLI

- Create a python-based CLI for running batch end-to-end pipelines. More details on the CLI are in [RTD](https://developer.skao.int/projects/ska-sdp-e2e-batch-continuum-imaging/en/latest/e2e_cli.html)

### Breaking

- Batchlet and batchlet plugins modules to new repository at https://gitlab.com/ska-telescope/sdp/ska-sdp-exec-batchlet.git.
- Remove streamlit sdp interface module.

## 0.4.2

### Batchlet

- Add a new option `dask_report_dir` which enables and stores dask performance reports. More details are in the [batchlet documentation](https://developer.skao.int/projects/ska-sdp-e2e-batch-continuum-imaging/en/latest/batchlet.html)

## 0.4.1

### Batchlet Plugins

- Read hostname from logs, and include it in the benchmon csv

### Batchlet

- Remove monitoring results for failed pipeline runs.


## 0.4.0

### Batchlet Plugins

- Added batchlet filter plugin for SKA SDP standard logging format
- Added consumer plugins for SKA SDP Configuration database (etcd) and simple csv file
- The plugins are part of the `ska_sdp_batchlet_plugins` package. Detailed documentation is available [here](https://developer.skao.int/projects/ska-sdp-e2e-batch-continuum-imaging/en/latest/batchlet_plugins.html)

### Batchlet

- Add support for log monitoring in batchlet
- Add support for adding custom filter and consumer plugins, used for monitoring of the logs
- **(Breaking change)** Update the `monitor` section of the input configuration schema. Please see the configuration details in the [batchlet documentation](https://developer.skao.int/projects/ska-sdp-e2e-batch-continuum-imaging/en/latest/batchlet.html)
- The `DaskSlurmCluster` class can take a decision appropriate parameters for the dask cluster based on allocated/available resources, similar to `LocalCluster`. Detailed documentation is available [here](https://developer.skao.int/projects/ska-sdp-e2e-batch-continuum-imaging/en/latest/dask_slurm_cluster.html)

### E2E Scripts

- Added [Rapthor](https://git.astron.nl/RD/rapthor) as part of e2e scripts
- The `e2e.sh` now calls other individual pipeline scripts as subprocesses.
- Added suppport for completely spack based execution, so user does not have to clone the scripts from git. The steps to run the scripts are updated in the [documentation](https://developer.skao.int/projects/ska-sdp-e2e-batch-continuum-imaging/en/latest/slurm_scripts.html)

## 0.3.2

### Batchlet

- DaskSlurmCluster now inherits `dask.distributed.SpecCluster` class. This allows DaskSlurmCluster to be used in manner similar to other clusters in dask ecosystem (`LocalCluster` of `SSHCluster`). This also adds ability to gracefully close the cluster upon exit.
- Add support for `ska-sdp-benchmark-monitor` in non-slurm (local) execution of `batchlet`.

### E2E Scripts

- Write output of instrumental calibration pipeline in the same directory as `OUTPUT_DIR` , instead of creating a unique timestamped subdirectory.

## 0.3.1

### Batchlet

- Force dask slurm cluster to start scheduler on random port, and use a unique scheduler file
- Allow batchlet to start a local cluster when slurm environment is not available
- Fix Typer's click dependency issue.

### E2E scripts

- Update slurm scripts to use spacks with metamodules

## 0.3.0

- Add `batchlet` tool to run batch pipelines. The documentation for batchlet is available on [readthedocs](https://developer.skao.int/projects/ska-sdp-e2e-batch-continuum-imaging/en/latest/batchlet.html).
- Removed`batch-continuum-imaging-pipeline` cli tool, it is now replaced by `batchlet`.
- The `ska_sdp_e2e_batch_continuum_imaging` package now contains slurm scripts, configuration and env setup files which are needed to run the end-to-end pipeline. Currently AWS environment is supported by default.


## 0.2.0

- Removed dependency between pipelines and end-to-end batch continuum imaging pipeline. 
- End-to-End batch continuum imaging pipeline now functions as a tool which runs pipeline(s) as a single job. Dependency management delegated to the upstream process.
- CLI Contract: End-to-End batch continuum imaging pipeline accepts a configuration file/stream and type of the configuration
- Use `srun` for DASK cluster creation instead of dask-jobqueues in-order to use the resources allocated to the slurm job.
- Add support for monitoring using [ska-sdp-benchmark-monitor](https://gitlab.com/ska-telescope/sdp/ska-sdp-benchmark-monitor)

## 0.1.0

Initial release of the SKA SDP end-to-end batch continuum imaging pipeline