Data Products
MCCS stores calibration solutions in the ArtefactRepository with the .h5 extension and deprecated .npy.
The .h5 contains all information the .npy does plus metadata.
Tip
There is a class ska_low_mccs.calibration_solver.CalibrationSolutionProduct
to help handle the .h5 files.
.npy data product
This contains just the solution with no metadata. All information saved here is also saved in the .h5 product.
Note
This is deprecated in favour of the .h5 product.
.h5 data product
This contains the solution together with metadata.
Each product has a HDF5 Attribute called structure_version used to
define that dataset.
Version 1.0
root
acquisition_time
corrcoeff
frequency_channel
galactic_centre_elevation
lst
masked_antennas
n_masked_final
n_masked_initial
residual_max
residual_std
solution
station_id
sun_adjustment_factor
sun_elevation
xy_phase
Database solution
Each solution consists of 2048 values (4 flattened complex numbers per antenna), this is stored in the calibration database together with metadata.
The PostgreSQL database schema is a single table (up to date)
tables:
- table: calibration_per_channel
columns:
- name: id
type: integer
description: Primary key identifier for each row.
- name: creation_time
type: timestamp without time zone
description: Time the record was stored in the database.
- name: frequency_channel
type: smallint
description: Frequency channel identifier.
- name: station_id
type: smallint
description: ID of the station associated with the solution.
- name: preferred
type: boolean
description: Whether this row is marked as the preferred solution.
- name: calibration_path
type: text
description: Path to the calibration/correlation matrix file.
- name: calibration_id
type: text
description: Unique calibration id.
- name: acquisition_time
type: double precision
description: >
Time the data was acquired (floating-point format).
- name: solution
type: ARRAY
description: >
The solution dataset. Ordered by TPM index:
``tpm_id * 16 + (adc_channel // 2)`` for 8 values per TPM.
- name: corrcoeff
type: ARRAY
description: Pearson correlation coefficients per polarisation.
- name: residual_max
type: ARRAY
description: >
Maximum absolute residual visibility deviation (per polarisation, in Kelvin).
Must be of length 4.
- name: residual_std
type: ARRAY
description: >
Standard deviation of residual visibilities (per polarisation, in Kelvin).
Must be of length 4.
- name: xy_phase
type: double precision
description: Estimated XY-phase error.
- name: n_masked_initial
type: integer
description: Number of masked antennas at the start.
- name: n_masked_final
type: integer
description: Number of masked antennas after calibration.
- name: lst
type: double precision
description: Local apparent sidereal time at the station.
- name: galactic_centre_elevation
type: double precision
description: Elevation of the Galactic Centre (in degrees).
- name: sun_elevation
type: double precision
description: Elevation of the Sun (in degrees).
- name: sun_adjustment_factor
type: double precision
description: Adjustment factor related to solar influence (unitless).
- name: masked_antennas
type: ARRAY
description: Antenna masked as bad. This is currently eep indexed. Length must be between 0 and 256 (inclusive).
- name: job_id
type: integer
description: The id of the job, this is a foreign key to calibration_job(id) cascading.
- name: solution_type
type: text
description: text representing the solution type.
- table: calibration_job
columns:
- name: id
type: integer
description: Primary key identifier for each row.
- name: sweep_id
type: integer
description: Time the record was stored in the database, this is a foreign key to frequency_sweep(sweep_id) cascading.
- name: initial_mask
type: ARRAY
description: Antennas masked from the spshw subsystem.
- name: calibration_parameters
type: json
description: ID of the station associated with the solution.
- name: user_friendly_name
type: text
description: A friendly name to identify this job. It can be something simple and autogenerated like "<stationid>_<date>_<time>".
- name: preferred
type: boolean
description: Whether this is the default calibration solution to use if one isn't chosen.
- table: frequency_sweep
columns:
- name: sweep_id
type: integer
description: Primary key identifing the sweep.
- name: station_id
type: smallint
description: The id of station this sweep is related to.
- name: channel_start
type: smallint
description: The starting channel
- name: channel_stop
type: smallint
description: The final channel
- name: path_to_data
type: text
description: path to data products.
- name: start_time
type: timestamp with time zone
description: Time sweep starts.
- name: nof_samples
type: integer
description: Number of samples.
- name: initial_static_delays
type: ARRAY
description: Static delays currently applied when sweep executed.
- table: calibration_per_sweep
columns:
- name: id
type: integer
description: Primary key identifying a calibration_per_sweep entry
- name: job_id
type: integer
description: foreign key 1 to 1 representing the calibration_job this calibration_per_sweep belongs to.
- name: final_mask
type: ARRAY
description: Masked antennas while generating this calibration_per_sweep solution.
- name: mean_amplitude
type: ARRAY
description: A single amplitude across the band per antenna
- name: phase
type: ARRAY
description: the phase offset at 0Hz per antenna per Jones element
- name: phase_gradient
type: ARRAY
description: the slope of the phase gradients per antenna per Jones element
- name: amplitude_fit_cost
type: double precision
description: Sum of squares scaled based on the number of points for amplitude.
- name: phase_fit_cost
type: ARRAY
description: Sum of squares scaled based on the number of points for phase per antenna per Jones element.
- name: phase_fit_covariance
type: ARRAY
description: the covariance on the phase fit per antenna per Jones element.