Changelog

1.3.0

Features

  1. Improve styling of the QA plots

Bug fixes

  1. Fix padding indices in delay calibration

  2. Ensure that delay_calibration stage does not apply the delay twice to the gaintable

  3. Fix export h5param function to make it more compatible with DP3

  4. Simplify ionospheric solver to only solve XX and YY polarisations

  5. SKB-1313: INST gain phase plots are not corrected for a reference antenna

1.2.1

  1. Update piper, sdp-func-python and sdp-datamodels dependencies

1.2.0

New Features & Enhancements

  1. Add SDM support for inst. This introduces the concept of qa_dir in stages planning on exporting diagnostic QA products. Field ID is used to differentiate MS files, and the same is used for obtaining sky model, and for exporting gaintables.

  2. Update skymodel reader with datamodels 1.3.0.

  3. Disable export visibilities stage by default.

Bug Fixes

  1. SKB-1311: Export crosspols if gain data is present.

  2. SKB-1312: Disable delay stage by default.

1.1.0

New Features & Enhancements

  1. Multiple-MS Support: INST can accept multiple measurement sets (multiple scans for the same field) as input. This is available for both calibrator INST and target INST pipelines.

  2. Updated the zernike_limit parameter to accept a list of integers instead of a single value.

Bug Fixes

  1. In flag_gain stage, resolve issues preventing the convergence of flagging cycle

  2. Fixed a bug in the load_data stage to handle antenna baseline order in correct way

1.0.0

Features

  1. Update ionospheric solver functions to solve based on solution intervals

Breaking

  1. The predict_visibility stage accepts config use_everybeam as boolean instead of beam_type as string.

  2. Use new LSM CSV Schema, adhereing to the data models from ska-sdp-datamodels.

General

  1. The modules which were deprecated in v0.8.0 are now removed. This includes some unused code, pipelines from PI25 (and corresponding notebooks).

0.8.0

Breaking

  1. Pass input visibilities (MSv2) as positional argument to the CLI instead of --input flag. Multiple inputs can be passed but only first one is used in pipelines. This change is made in all of the CLI pipelines.

  2. Use SKA LSM CSV format instead of OSKAR csv format as input sky model.

  3. Remove mode config parameter from flag_gain stage. Fix the mode basis of soltype.

  4. Remove solver config parameter from complex_gain_calibration stage used for instrumental target calibration. Use gain_substitution as default solver.

General

  1. Use AA0.5 strategy for benchmarking configuration.

  2. Move processing and solver functions to ska-sdp-func-python library.

  3. Add end-to-end integration test for INST CLI application

Fixes

  1. Fix issue with delay values being misinterpreted as RGB

  2. Fix phase wrapping in gain flagging stage. Solution type is internally mapped with mode.

0.7.1

  1. Remove normalise_gains option from solver configuration

  2. Add support for numpy 2+

  3. Fix optional stages being disabled issue, for experimental subcommand

0.7.0

New features

  1. Filtering of the visibility data based on baselines and uv distance values, before bandpass calibration starts

  2. Flagging based on real and imaginary parts of the gaintable

  3. Option to export the filtered sky model used for the prediction of visibilities, in a CSV file format.

Configuration Changes

Stages

  1. INST pipeline's default stage order is updated as per the AA0.5 strategy. Some of the stages are turned off by default.

  2. A new bandpass_initialisation stage is added to initialize the gaintable before bandpass calibration.

Additions

  1. export_sky_model in predict_vis : Option to export the filtered sky model

  2. element_response_model in predict_vis : An option from everybeam to select the desired element response model.

  3. visibility_filters in bandpass_calibration : Options to filter visibilities before bandpass calibration starts

Removals

  1. eb_coeffs from predict_vis: It was used to pass path to everybeam coefficients to everybeam library. If coefficients must be passed, then the user must set EVERYBEAM_DATADIR environment variable before running the INST pipeline. If using spack package of everybeam, then EVERYBEAM_DATADIR is automatically set by spack after loading.

  2. soltype in flag_gain : Value "both" is removed from allowed values. New allowed values are: "phase", "amplitude", "amp-phase", "real-imag"

  3. timeslice from run_solver_config : For instrumental calibration, this parameters was unused. For complex gain calibration, the timeslice is passed via the target_load_data stage.

Refactoring / Improvements

  1. All functions have been re-organised into following packages:

    1. "numpy_processors": Work on numpy arrays and return numpy arrays

    2. "xarray_processors": Work on xarray dataarray/dataset instances (based on ska-sdp-datamodels).

  2. Made predict_vis stage parallelizable across both time and frequency chunks

  3. Distribute the rotation measure calculation across multiple dask tasks

0.6.0

Added

  • Ionospheric Delay calibration for calibrator.

  • DI calibration for target:

    • Complex Gain Calibration

    • Ionospheric Delay Calibration

Improved

  • Dask computation for compute and checkpoint tasks.

  • Refactored plot gaintable.

0.5.0

Minor

  • Include script to apply gains and generate fits image using wsclean and dp3

  • Use XX = YY = 2I convention for predicting visibility from skycomponents

  • Flagging based on the gain values

  • Fix bug with plot labeling.

  • Include pipeline validation notebooks for stage combinations

0.4.1

Patch

  • Fixed bug with incorrect beams computation

0.4.0

Breaking

  • The configuration schema (YAML) has changed for many stages. Some notable changes:

    • load_data stage now has parameters corresponding to the conversion from MSv2 to Zarr, like nchannels_per_chunk, ntimes_per_ms_chunk, cache_directory.

    • Ineffective parameters like reset_vis, jones_type, export_model_vis, flagging are removed.

    • fchunk parameter is removed, as we expect the entire pipeline to work with consistent chunksizes for all dimensions, from start till finish.

    Please refer to the stage_config page.

  • For distributed run using dask workers, the workers must have a resource called process. Please refer to the "dask distribution" section in README to understand the usage.

Added

  • Support providing antenna names (along with indices) for config parameters which refer to a antenna, like refant or station

Improvements

Reducing memory footprint by using zarr as input

  • The pipeline will first convert the input MSv2 into a zarr file, which represents the Visibility data model.

  • The zarr file will be chunked across frequency and time dimensions based on the parameters to load_data stage.

  • Xarray operations like map_blocks work well with zarr format, minimising data which is loaded at a time to memory.

  • The intermediate zarr files will be cached based on the name of MSv2, field id and data description id; and stored in user provided cache_directory. This will ensure that cached zarr files are re-used between multiple runs on the same input MSv2.

0.3.4

Fixed

  • Fixed issue with dask.array.float32 targeted at spack builds.

0.3.3

Added

  • Apply gaintable on visibility.

  • Export corrected visibility and model visibility.

0.3.0

Added

  • Added Faraday Rotation into modular INST Pipeline.

  • Introduced export_visibility stage which currently supports applying gaintable on visibilities.

  • Added smooth gain solutions stage.

  • Allow exporting intermediate gain solutions as h5parm file.

Changed

  • Improved differential Faraday rotation fits in model_rotations and associated updates to bandpass_polarisation.

  • Made load_data stage as reorderable.

  • Made model rotation dask compatible.

  • Input configuration file schema is changed slightly for generate_channel_rm stage.

Fixed

  • Logging issue in LSM generation.

0.2.2

Added

  • Make metadata generation optional.

0.2.1

Added

  • Changed the way everybeam is called and added normalisation for OSKAR datasets.

  • Functionality to use a csv file to generate the sky model.

  • CLI installable using Piper.

  • Instrument calibration pipeline as a collection of stages.

  • Delay calibration stage.

  • Ability to reorder stages.

  • Generation of calibration plots.

  • Generation of metadata file along with data products.

  • Allow configuring csv based custom components.

  • The way everybeam models are initialised for Low datasets has been updated.

  • Normalisation has been simplified and updated for OSKAR simulations.

Fixed

  • Bug in pipeline_config for parameter fov.

  • Bug in deconvolve_gaussian for circular Gaussian components.

0.1.6

Added

  • Discard unused polarisation dimensions before writing H5Parm file.

  • H5Parm calibration solution output.

0.1.5

Fixed

  • After gaintable creation, run_solver resets the gaintable interval to include all times. This avoids a bug in create_gaintable_from_visibility.

Changed

  • Forced a single polarisation chunk during load_ms, to be consistent with other dimensions.

0.1.4

Changed

  • Forced a single time chunk during load_ms.

0.1.3

Fixed

  • Documentation badge.

0.1.2

Changed

  • LSM Component elliptical Gaussian parameter names have been updated.

  • LSM Component parameter Fint200 has been replaced with flux and ref_freq.

  • Pipeline argument "dask_cluster" replaced with "dask_scheduler_address", which accepts the cluster IP rather the the object.

  • Station-dependent beam models are extracted from EveryBeam.

  • Simulations and tests have been reduced to AA1 (from AA2).

0.1.1

Added

  • PipelineConfig class for pipelines.

  • Pipelines can accept a user defined-dask cluster.

  • Default values for a number of lsm Component variables.

  • More options in create_demo_ms.

  • .readthedocs.yaml file.

Changed

  • Pipelines can be called with a user-defined local sky model and the option of not using a beam model.

  • Pipeline parameter eb_ms defaults to the input measurement set.

  • A single baseline chunk is set in load_ms. This dimension requires modification and auto chunking can cause confusion.

  • Pre-define work array in predict_from_components to avoid memory leak build up.

  • Improvements to dask handling in dask_wrappers.

0.1.0

Added

  • Demo pipelines and notebooks. Documentation and unit tests.

  • Functions to handle data-model confusion during xarray dask operations.

  • Support for elliptical Gaussian sky components during predict.

  • General calibration tasks with ask-enabled ingest, predict, solve and apply options.

  • Pulled in content from ska-python-skeleton.

  • Empty Python project directory structure.