.. _data_products: Data products of the pointing pipeline ====================================== The pointing pipeline outputs several files, written to disc, for further analysis. These data products are described here. HDF5 main data product ---------------------- The main data product of the pointing pipeline is an `HDF5 `_ file based on the :external+ska-sdp-datamodels:class:`~ska_sdp_datamodels.calibration.PointingTable` data model. One file per pointing observation is generated. The file is set up to provide all relevant information for individual reference pointing observations, as well as the inputs required to generate an offline global pointing model. For data with 61 dishes and 4096 frequency channels, the HDF5 output file for a single reference pointing observation is 28kB in size. HDF5 format is a self describing data format containing metadata (attributes) and binary data organised in a "file directory" like structure. The detailed format of the pointing output product is shown in the tables below. Each file contains the results from one pointing observation - i.e. a single :external+ska-sdp-datamodels:class:`~ska_sdp_datamodels.calibration.PointingTable` object in the HDF5 group labelled ``"PointingTable0"``. Product quality ^^^^^^^^^^^^^^^ The HDF5 file contains the fitted offsets and their uncertainties. The fitted offsets must pass a number of quality criteria (see :ref:`fitting_qa`) in order to be written into the output. If these criteria are not met, the output offsets are set to ``NaN``. Accessing and reading the files ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Details of how to locate and read the HDF5 files are described in a `Jupyter notebook `_. Global attributes ^^^^^^^^^^^^^^^^^ .. list-table:: :header-rows: 1 * - Attribute Name - Type - Description * - number_data_models - long - Number of data models contained in file Group attributes ^^^^^^^^^^^^^^^^ .. list-table:: :header-rows: 1 * - Attribute Name - Type - Description * - data_model - string - Python data model class, i.e. :external+ska-sdp-datamodels:class:`~ska_sdp_datamodels.calibration.PointingTable` * - pointing_frame - string - Frame of reference used for pointing fits, e.g. ``"xel-el"`` * - pointingcentre_coords - string - Pointing centre RA/Dec coordinates in degrees * - pointingcentre_frame - string - Coordinate frame for pointing centre, e.g. ``"icrs"`` * - receptor_frame - string - Polarisation state, e.g. ``"StokesI"`` * - band_type - string - Observing band, e.g. ``"Band 2"`` * - scan_mode - string - Pointing observation mode, e.g. ``"5-point"`` * - track_duration - double - How long each scan position was tracked for in seconds. Multiple pointing timestamps are required to calculate the integration time and hence the ``track_duration``. If only one pointing timestamp is present, ``None`` is stored as the track duration * - discrete_offset - double - Discrete offset of each scan from the central scan, in degrees (xel-el) * - commanded_pointing - double - Commanded pointing of each dish at the middle timestamp of the central scan of the pointing observation (radians) .. Group data ^^^^^^^^^^ Dimensions: - **nTime**: number of time chunks (currently=1) - **nAnt**: number of antennas - **nFreq**: number of frequency chunks (currently=1) - **nRecep**: number of polarisations (currently=1, "StokesI") - **nCoord**: number of coordinate angles in pointing frame - i.e. 2 for (xel, el) .. list-table:: :header-rows: 1 * - Name - Type (dimensions) - Unit - Description * - configuration - sub-group - - Data model for telescope configuration containing antenna characteristics following the `MSv3`_ definition (see below for full description). * - data_expected_width - double (nTime, nAnt, nFreq, nRecep, nCoord) - rad - The theoretical voltage beam sizes for all antennas in radians in the horizontal and vertical co-polarisations * - data_fitted_height - double (nTime, nAnt, nFreq, nRecep) - - The fitted Gaussian height for all antennas in arbitrary units * - data_fitted_height_std - double (nTime, nAnt, nFreq, nRecep) - - The standard error on the fitted_height in arbitrary units * - data_fitted_width - double (nTime, nAnt, nFreq, nRecep, nCoord) - rad - The fitted voltage beam sizes for all antennas in the horizontal and vertical co-polarisations * - data_fitted_width_std - double (nTime, nAnt, nFreq, nRecep, nCoord) - rad - The standard error on the fitted_width * - data_frequency - double (nFreq) - Hz - The central frequency in Hz if fitting to gains and frequency at the higher end of the band if fitting to visibilities * - data_pointing - double (nTime, nAnt, nFreq, nRecep, nCoord) - rad - The pointing offsets in cross-elevation and elevation in radians for all antennas * - data_time - double (nTime) - s - The time in MJD at the middle of all the scans * - data_weight - double (nTime, nAnt, nFreq, nRecep, nCoord) - rad\ :sup:`-2` - The inverse square of the standard error in the fitted pointing values Configuration sub-group """"""""""""""""""""""" Data model for telescope configuration containing antenna characteristics following the `MSv3`_ definition. Note: the sub-group data are accessed via nested sub-subgroup also named "configuration". .. list-table:: Configuration sub-group attributes :header-rows: 1 * - Attribute Name - Type - Description * - data_model - string - Python data model class, i.e. :external+ska-sdp-datamodels:class:`~ska_sdp_datamodels.configuration.Configuration` * - name - string - Telescope name * - location - string - xyz location in m * - frame - string - Reference coordinate frame for location * - receptor_frame - string - Polarisation state, e.g. "StokesI" .. list-table:: Configuration sub-group data :header-rows: 1 * - Name - Type (dimensions) - Unit - Description * - diameter - double (nAnt) - m - Dish diameter * - mount - string (nAnt) - - Mount type of the antenna eg. "azel" * - names - string (nAnt) - - Dish names * - offset - double (nAnt, 3) - m - Axes offset of mount to feed reference point * - stations - string (nAnt) - - Station names * - vp_type - string (nAnt) - - Voltage pattern type eg "MID" * - xyz - double (nAnt, 3) - m - Antenna positions in Earth-Centred Earth Fixed coordinates .. _metadata_yaml: Metadata yaml file ------------------ Each HDF5 file has a corresponding yaml file that contains additional metadata items. The yaml metadata file contains: - Execution block ID - The context of the execution block provided by the Observation Execution Tool (OET) - The configuration of the processing block used to generate the data products - A list of data product files with description, status, size and a cyclic redundancy check (CRC) for error detection - Associated `IVOA `_ ObsCore attributes used for querying astronomical observations The `IVOA `_ ObsCore attributes currently implemented for pointing files are listed in the table below. .. list-table:: :header-rows: 1 * - Metadata name - Type - Unit - Description * - dataproduct_type - String - - Data type * - calib_level - Integer - - Calibration level (0, 1, 2, 3, 4) * - obs_collection - String - - Name of the data collection * - obs_id - String - - Observation ID * - access_format - String - - File content format * - access_estsize - integer - kB - Estimated size of dataset * - target_name - String - - Astronomical object observed, if any * - s_ra - double - deg - Central right ascension, ICRS * - s_dec - double - deg - Central declination, ICRS * - t_min - double - s - Start time in MJD * - t_max - double - s - Stop time in MJD * - em_min - double - m - Start wavelength * - em_max - double - m - Stop wavelength * - pol_states - String - - List of polarization states or NULL if not applicable * - pol_xel - integer - - Number of polarization samples * - facility_name - String - - Name of the facility used for this observation * - instrument_name - String - - Name of the instrument used for this observation Additional outputs ------------------ TXT file ^^^^^^^^ A CSV-type text file is generated with the following header:: AntennaName, CrossElevationOffset, CrossElevationOffsetStd, ElevationOffset, ElevationOffsetStd, ExpectedWidthH, ExpectedWidthV, FittedWidthH, FittedWidthHStd, FittedWidthV, FittedWidthVStd, FittedHeight, FittedHeightStd For the description of each column, see the corresponding data item in the HDF5 file description above. However, note that contrary to the HDF5 file, the cross-elevation and elevation offsets have their standard deviations listed in the text file, and not the weight values. `H` stands for horizontal, `V` stands for vertical co-polarisation. PNG files ^^^^^^^^^ The pipeline (when run in standalone mode) produces various plots for validation purposes. - Visibility amplitudes: The visibility amplitudes for the parallel-hands polarisations when fitting the beams to the visibility amplitudes option is used - Gain amplitudes: The un-normalised gain amplitudes for all antennas across the entire frequency band or sub-bands when fitting the beams to the gain amplitudes option is used - On-sky offsets: The commanded pointings relative to the source in cross-elevation and elevation in the reference pointing observation - Gaussian fits: The Gaussian fit in cross-elevation and elevation showing the fitted centre across the entire frequency band or in all sub-bands - Pointing offsets: The weighted average of the pointing offsets shown in the Gaussian fits, other fitted parameters, and their uncertainties .. _MSv3: https://casacore.github.io/casacore-notes/264.html#x1-220003