Changelog

1.0.2

  • Refactored CLI app: use argparse instead of docopt.

  • Refactored top-level pipeline classes: they are now pydantic-based with improved parameter validation.

  • Use cached_property decorator instead of manual caching in places where it was safe to update the code.

  • Make the standalone pipeline end-to-end test runnable locally, and make it run systematically as part of the test suite. Added the test MeasurementSet to tests/data with a README file.

Fixed

  • The –msdir CLI argument does not need to contain a trailing slash anymore.

1.0.1

Changed

  • Optimised Docker image build process for development.

  • Updated to latest python Docker base images.

  • Codeowners updated to Yabby Team members (MR147)

Fixed

  • Data product flow transitions to correct end states FAILED and COMPLETED instead of FAULT and COMPLETE.

  • Path resolution for vis-receive Measurement Sets now works against recent vis-receive releases. Resolution is now performed by inspecting the vis-receive dataproduct flow state where paths are directly published.

1.0.0

  • Fix loading MS path information from vis-receive data flow (MR146)

  • Update documentation (MR145)

  • Update dependencies, including:

    • ska-sdp-datamodels: 1.0.0

    • ska-sdp-dataqueues: 1.0.0.post1

    • ska-sdp-dataproduct-metadata: 1.0.0.post1

    • ska-sdp-func-python: 1.0.0.post1

    • ska-telmodel: 1.20.0

    (MR144, MR145)

  • Updates the uncertainty calculation after the fitted parameters are averaged (MR140)

  • Data flow state status values are now FAULT or COMPLETE (MR143)

  • Data flow state “internal” is now a list of lists (MR143)

  • Bug fix in updating state of data flows in SDP (MR142)

  • Bug fix in writing metadata file when running pipeline in SDP (MR136)

  • Add documentation on how averaging is computed (MR141)

  • Simplify beam fitting methods (MR138)

  • Compute weighted standard deviation instead of sum of weights in the time-and-polarisation averaging function (MR139)

  • Update Dockerfile to use SKA Python base image (MR132)

  • Update pointing pipeline to monitor multiple pointing scans(MR134)

  • Update documentation for users to use pipeline with minimal or no support (MR133)

  • Split visibility into frequency chunks before applying static RFI mask(MR131)

0.9.0

  • Update pointing offset pipeline to obtain MS directory from flow source (MR127)

  • Update usage of send via data queue with version ska-sdp-dataqueues 0.4.0 (MR128)

0.8.0

  • Remove generating plots when running the pipeline in SDP (MR126)

  • Add back Gaussian fits plot in two-dish standalone mode (MR125)

  • Add reporting processing errors in the data flow state (MR123)

  • Ensure dish names and their corresponding fitted pointing offsets written into the PointingTable and CSV are ordered correctly, and they match (MR122)

  • Update usage of DataQueue with version ska-sdp-dataqueues 0.3.0 (MR121)

  • Remove providing empty “interval” and “residual” parameters in PointingTable (using latest ska-sdp-datamodels) (MR120)

  • Update ska-sdp-dataqueues==0.2.0, ska-sdp-datamodels==0.3.3 (MR120)

  • Add track_duration to PointingTable (MR119)

  • Re-structure Gaussian fits plot to handle beam fits to parallel-hands gain amplitudes (MR82)

  • Update documentation for the option to fit the primary beams to the parallel-hands gain amplitudes (MR82)

  • Added the option to fit the primary beams to the parallel-hands gain amplitudes (MR82)

0.7.0

  • Bug fix when calculating on-sky-offsets from TARGET column (MR116)

  • Use data flow entries to determine where to store data products (MR115)

  • Using data flow entry in Config DB to get Kafka queue information (MR110)

  • Use the Data Queue library to pass the pointing offset results to Kafka (MR112)

0.6.0

  • Fix error in plotting the Gaussian fits because of zero gain amplitudes for an antenna for all scans (MR113)

  • Update tests for fitting beams to visibility amplitudes (MR111)

  • Discard zero gain amplitudes and amplitudes close to approximately zero on-sky offsets (MR101)

  • Improvements to documentation (MR107, MR108)

  • Make sure data are sent with dish IDs to Kafka, even if pipeline fails (MR106)

  • Move convert_to_structured function to utils.py (MR105)

  • Set up configurable logging and remove log duplication (MR104)

  • Add reference time and corresponding commanded azel when fitting to the visibility amplitudes (MR102)

  • Introduce InitPipeline, SDPPipeline, and StandAlonePipeline classes to reduce CLI complexity (MR81)

  • Reduce read_data.py complexity (MR99)

  • A single HDF5 file, text file, and offset plot is output in the two-dish scenario (MR95)

  • The beams are fitted to the moving antenna in a two-dish scenario (MR95)

  • Minor restructuring of the CLI (MR95)

  • Fix phase centre bug and failing test with data (MR98)

  • Refactor plotting and output functions to adapt to two-dish mode (MR94)

  • Refactor plotting functions (MR93)

  • Add documentation for the two-dish mode scenario (MR92)

  • Refactor documentation (MR91)

  • Add functionality to split the MS files for two-dish scan modes (MR89)

0.5.0

  • Document the data products and unify variable names (MR90)

  • Add reference time and corresponding commanded azel (MR85, bugfix: MR86)

  • Fixing corrupted pointing table (MR84)

  • Updated to send data to Kafka as a structured numpy data. Also include the index of the final scan in this data (MR83)

  • Updated to publish pointing offset results in degrees for kafka (MR80)

  • Add exception handler for plotting functions(MR79)

  • Generate a smaller RFI mask from an existing larger one in the telmodel store (MR77)

  • Add support for visibility fitting per sub-band (MR77)

0.4.0

  • Remove need for extra ghost scan in pointing processing (MR67)

  • Add ska-sdp-func as optional dependency (MR71)

  • Added extra metadata to PointingTable (MR68)

  • Save data and metadata file into a scan-based directory structure (MR74)

  • Load RFI mask in HDF5 format from the telescope model data repository (MR69)

  • Update the metadata saved in ska-data-product.yaml with more Obscore items (MR73)

  • Update the URL containing the test datasets after reorganisation of data on GCP (MR72)

  • Fixed a bug where the pointing and visibility time sampling were assumed to be the same and Gaussian fits plot bugs (MR70)

  • Handle errors generated by pipeline and store information in processing blocks state (MR64)

  • Allow pipeline to read and write processed entry into processing block (MR61)

  • Pointing results saved automatically and removed some command line arguments (MR63)

  • Bug fix in reading pointing timestamps from the TIME column of the pointing sub-table (MR62)

  • Refactor pipeline to output HDF5 data product (MR57)

  • Update CI test jobs to uses standard make targets (MR58)

  • All parameters in the output text file are now written into the updated PointingTable (MR56)

  • Added option to send pointing results to Kafka in JSON format (MR55)

  • Fixed shape of antenna pointings read from pointing sub-table of Measurement Sets (MR54)

  • Added option to read on-sky offsets in cross-elevation and elevation from SOURCE_OFFSET column of the pointing sub-table (MR47)

0.3.0

  • Added functions to process multiple scan batches (MR52)

  • Plots fitted parameters with errorbars (MR46)

  • Added skart toml file which handles dependencies in SKA repositories (MR45)

  • Bug fixes, documentation and Gaussian plot improvements (MR44)

  • Added option to use model visibility in the gain solver (MR43)

  • Converts discrete offset pointings in azimuth to cross-elevation before fitting (MR42)

  • Update read_data to support MJD in pointing table (MR39)

0.2.0 (same as 0.1.1)

  • Added end-to-end test that uses MeerKAT AA0.5 data (MR35)

  • Improved each scan’s pointing timestamps in the tests (MR34)

  • Antenna pointings read from Measurement Sets are in radians (MR32)

  • Fixed documentation build for pipeline (MR31)

  • Update logic to obtain pointing scan IDs from the Configuration Database (MR30)

  • Added function to send data to Kafka queues (MR28, MR36)

  • Updated pipeline to read data from multiple Measurement Sets Based on Scan ID (MR27)

  • Used PointingTable to output pointing offsets (MR26)

  • Using zeroth-order spline interpolation to align pointing and visibility datasets (MR25)

  • Added monitoring of execution blocks for scan type and id (MR24)

  • Added plotting functions (MR23)

  • Added support for gain calibration per sub-band (MR22)

  • Added option to use calibration weights when fitting primary beams to the gain amplitudes (MR21)

  • Bug fix in accessing the concatenated antenna pointings (MR20)

  • Added functionality to read multiple Measurement Sets, one per discrete pointing scan (MR19)

0.1.0

  • Added Dockerfile for docker image build (MR18)

  • Fixed reshaping of cross-correlation visibilities when using the fit_to_vis option (MR17)

  • Added function to interpolate timestamps from data (MR16)

  • Added functionality for fitting the primary beams to the gain amplitudes (MR14)

  • Added beamwidth factor parameter (MR13)

  • Modified function to read RFI mask file to read data from txt (MR12)

  • Removed the use of rdb option as parameters are read from the Measurement Set (MR10)

  • Updated documentation for the pipeline (MR9)

  • Added integration test for the pipeline (MR8)

  • Added functionality for fitting the primary beams to the visibility amplitudes (MR7)

  • Created documentation for the pipeline (MR5)

  • Added functions for reading Measurement Set and writing pointing offsets to text file (MR3)

  • Added supporting functions for filtering data and coordinates transformation (MR2)

  • Initial setup of repository (MR1, MR4)