# Changelog

## 1.3.0

### Features

1. Improve styling of the QA plots

### Bug fixes

1. Fix padding indices in delay calibration
2. Ensure that delay_calibration stage does not apply the delay twice to the gaintable
3. Fix export h5param function to make it more compatible with DP3
4. Simplify ionospheric solver to only solve XX and YY polarisations
5. SKB-1313: INST gain phase plots are not corrected for a reference antenna

## 1.2.1

1. Update piper, sdp-func-python and sdp-datamodels dependencies	
	
## 1.2.0

### New Features & Enhancements

1. Add SDM support for inst. This introduces the concept of _qa_dir_ in stages planning on exporting diagnostic QA products. Field ID is used to differentiate MS files, and the same is used for obtaining sky model, and for exporting gaintables.
2. Update skymodel reader with datamodels 1.3.0.
3. Disable export visibilities stage by default.

### Bug Fixes

1. SKB-1311: Export crosspols if gain data is present.
2. SKB-1312: Disable delay stage by default.

## 1.1.0

### New Features & Enhancements

1. Multiple-MS Support: INST can accept multiple measurement sets (multiple scans for the same field) as input. This is available for both calibrator INST and target INST pipelines.

2. Updated the zernike_limit parameter to accept a list of integers instead of a single value.

### Bug Fixes

1. In `flag_gain` stage, resolve issues preventing the convergence of flagging cycle

1. Fixed a bug in the `load_data` stage to handle antenna baseline order in correct way

## 1.0.0

### Features

1. Update ionospheric solver functions to solve based on solution intervals

### Breaking

1. The `predict_visibility` stage accepts config `use_everybeam` as boolean instead
 of `beam_type` as string.
1. Use new LSM CSV Schema, adhereing to the data models from `ska-sdp-datamodels`.

### General

1. The modules which were deprecated in v0.8.0 are now removed. This includes some unused code, pipelines from PI25 (and corresponding notebooks).

## 0.8.0

### Breaking

1. Pass input visibilities (MSv2) as positional argument to the CLI instead of `--input` flag. Multiple inputs
 can be passed but only first one is used in pipelines. This change is made in all of the CLI pipelines.
2. Use SKA LSM CSV format instead of OSKAR csv format as input sky model.
3. Remove `mode` config parameter from `flag_gain` stage. Fix the mode basis of soltype.
4. Remove `solver` config parameter from complex_gain_calibration stage used for instrumental target calibration.
 Use `gain_substitution` as default solver. 

### General

1. Use AA0.5 strategy for benchmarking configuration.
2. Move processing and solver functions to ska-sdp-func-python library.
3. Add end-to-end integration test for INST CLI application

### Fixes

1. Fix issue with delay values being misinterpreted as RGB
2. Fix phase wrapping in gain flagging stage. Solution type is internally mapped with mode.

## 0.7.1

1. Remove `normalise_gains` option from solver configuration

2. Add support for numpy 2+

3. Fix optional stages being disabled issue, for experimental subcommand

## 0.7.0

### New features

1. Filtering of the visibility data based on baselines and uv distance values, before bandpass calibration starts

2. Flagging based on real and imaginary parts of the gaintable

3. Option to export the filtered sky model used for the prediction of visibilities, in a CSV file format.

### Configuration Changes

#### Stages

1. INST pipeline's default stage order is updated as per the AA0.5 strategy. Some of the stages are turned off by default.

2. A new `bandpass_initialisation` stage is added to initialize the gaintable before bandpass calibration.

#### Additions

1. `export_sky_model` in `predict_vis` : Option to export the filtered sky model

2. `element_response_model` in `predict_vis` : An option from everybeam to select the desired element response model.

3. `visibility_filters` in `bandpass_calibration` : Options to filter visibilities before bandpass calibration starts

#### Removals

1. `eb_coeffs` from `predict_vis`: It was used to pass path to [everybeam coefficients](https://git.astron.nl/RD/EveryBeam/-/tree/master/coeffs) to `everybeam` library. If coefficients must be passed, then the user must set `EVERYBEAM_DATADIR` environment variable before running the INST pipeline. If using spack package of `everybeam`, then `EVERYBEAM_DATADIR` is automatically set by spack after loading.

2. `soltype` in `flag_gain` : Value "both" is removed from allowed values. New allowed values are: "phase", "amplitude", "amp-phase", "real-imag"

3. `timeslice` from `run_solver_config` : For instrumental calibration, this parameters was unused. For complex gain calibration, the `timeslice` is passed via the `target_load_data` stage.


### Refactoring / Improvements

1. All functions have been re-organised into following packages:
   1. "numpy_processors": Work on numpy arrays and return numpy arrays
   2. "xarray_processors": Work on xarray dataarray/dataset instances (based on ska-sdp-datamodels).

2. Made `predict_vis` stage parallelizable across both time and frequency chunks

3. Distribute the rotation measure calculation across multiple dask tasks

## 0.6.0

### Added

- Ionospheric Delay calibration for calibrator.
- DI calibration for target:
  - Complex Gain Calibration
  - Ionospheric Delay Calibration

### Improved

- Dask computation for compute and checkpoint tasks.
- Refactored plot gaintable.

## 0.5.0

### Minor

- Include script to apply gains and generate fits image using `wsclean`
  and `dp3`
- Use XX = YY = 2I convention for predicting visibility from
  skycomponents
- Flagging based on the gain values
- Fix bug with plot labeling.
- Include pipeline validation notebooks for stage combinations

## 0.4.1

### Patch

- Fixed bug with incorrect beams computation

## 0.4.0

### Breaking

- The configuration schema (YAML) has changed for many stages. Some
  notable changes:

  - `load_data` stage now has parameters corresponding to the conversion
    from MSv2 to Zarr, like `nchannels_per_chunk`,
    `ntimes_per_ms_chunk`, `cache_directory`.
  - Ineffective parameters like `reset_vis`, `jones_type`,
    `export_model_vis`, `flagging` are removed.
  - `fchunk` parameter is removed, as we expect the entire pipeline to
    work with consistent chunksizes for all dimensions, from start till
    finish.

  Please refer to the `stage_config` page.

- For distributed run using dask workers, the workers must have a
  resource called `process`. Please refer to the "dask distribution"
  section in [README](README) to understand the usage.

### Added

- Support providing antenna names (along with indices) for config
  parameters which refer to a antenna, like `refant` or `station`

### Improvements

#### Reducing memory footprint by using zarr as input

- The pipeline will first convert the input MSv2 into a zarr file, which
  represents the `Visibility` data model.
- The zarr file will be chunked across frequency and time dimensions
  based on the parameters to `load_data` stage.
- Xarray operations like `map_blocks` work well with zarr format,
  minimising data which is loaded at a time to memory.
- The intermediate zarr files will be cached based on the name of MSv2,
  field id and data description id; and stored in user provided
  `cache_directory`. This will ensure that cached zarr files are re-used
  between multiple runs on the same input MSv2.

## 0.3.4

### Fixed

- Fixed issue with dask.array.float32 targeted at spack builds.

## 0.3.3

### Added

- Apply gaintable on visibility.
- Export corrected visibility and model visibility.

## 0.3.0

### Added

- Added Faraday Rotation into modular INST Pipeline.
- Introduced export_visibility stage which currently supports applying
  gaintable on visibilities.
- Added smooth gain solutions stage.
- Allow exporting intermediate gain solutions as h5parm file.

### Changed

- Improved differential Faraday rotation fits in model_rotations and
  associated updates to bandpass_polarisation.
- Made load_data stage as reorderable.
- Made model rotation dask compatible.
- Input configuration file schema is changed slightly for
  generate_channel_rm stage.

### Fixed

- Logging issue in LSM generation.

## 0.2.2

### Added

- Make metadata generation optional.

## 0.2.1

### Added

- Changed the way everybeam is called and added normalisation for OSKAR
  datasets.
- Functionality to use a csv file to generate the sky model.
- CLI installable using Piper.
- Instrument calibration pipeline as a collection of stages.
- Delay calibration stage.
- Ability to reorder stages.
- Generation of calibration plots.
- Generation of metadata file along with data products.
- Allow configuring csv based custom components.
- The way everybeam models are initialised for Low datasets has been
  updated.
- Normalisation has been simplified and updated for OSKAR simulations.

### Fixed

- Bug in pipeline_config for parameter fov.
- Bug in deconvolve_gaussian for circular Gaussian components.

## 0.1.6

### Added

- Discard unused polarisation dimensions before writing H5Parm file.
- H5Parm calibration solution output.

## 0.1.5

### Fixed

- After gaintable creation, run_solver resets the gaintable interval to
  include all times. This avoids a bug in
  create_gaintable_from_visibility.

### Changed

- Forced a single polarisation chunk during load_ms, to be consistent
  with other dimensions.

## 0.1.4

### Changed

- Forced a single time chunk during load_ms.

## 0.1.3

### Fixed

- Documentation badge.

## 0.1.2

### Changed

- LSM Component elliptical Gaussian parameter names have been updated.
- LSM Component parameter Fint200 has been replaced with flux and
  ref_freq.
- Pipeline argument "dask_cluster" replaced with
  "dask_scheduler_address", which accepts the cluster IP rather the the
  object.
- Station-dependent beam models are extracted from EveryBeam.
- Simulations and tests have been reduced to AA1 (from AA2).

## 0.1.1

### Added

- PipelineConfig class for pipelines.
- Pipelines can accept a user defined-dask cluster.
- Default values for a number of lsm Component variables.
- More options in create_demo_ms.
- .readthedocs.yaml file.

### Changed

- Pipelines can be called with a user-defined local sky model and the
  option of not using a beam model.
- Pipeline parameter eb_ms defaults to the input measurement set.
- A single baseline chunk is set in load_ms. This dimension requires
  modification and auto chunking can cause confusion.
- Pre-define work array in predict_from_components to avoid memory leak
  build up.
- Improvements to dask handling in dask_wrappers.

## 0.1.0

### Added

- Demo pipelines and notebooks. Documentation and unit tests.
- Functions to handle data-model confusion during xarray dask
  operations.
- Support for elliptical Gaussian sky components during predict.
- General calibration tasks with ask-enabled ingest, predict, solve and
  apply options.
- Pulled in content from ska-python-skeleton.
- Empty Python project directory structure.