PST metadata mapping

The following sections record the mapping between the scan configuration and scan request to the header keys within the output DADA files during PST digital signal processing. It also includes the mapping to the SDP’s data product dashboard (DPD) metadata keys.

General mapping

This section includes the mapping from the scan configuration, DADA Header keys, and direct mappings to the DPD metadata. Where this is a simple conversion or calculation then the information is in the the Notes column. The Mode column is for the observation mode:

  • VR - Voltage recorder

  • FT - Flow through

  • DS - Dynamic spectrum (currently not supported)

  • PT - Pulsar timing (currently not supported)

  • All - all modes

Config Key / Default DADA Header Key DPD Key Notes
eb_id EB_ID execution_block
scan_id SCAN_ID obscore/obs_id Scan ID comes from the scan not the scan configuration
udp_nsamp UDP_NSAMP
wt_nsamp WT_NSAMP
udp_nchan UDP_NCHAN
receiver_id FRONTEND
feed_polarization FD_POLN
feed_handedness FD_HAND
feed_angle FD_SANG
feed_tracking_mode FD_MODE
feed_position_angle FA_REQ
receptors ANTENNAE input is an array of values converted to string and joined by a ","
receptor_weights ANT_WEIGHTS input is an array of values converted to string and joined by a ","
num_of_polarizations NPOL obscore/pol_xel For SKA should be 2
bits_per_sample NBIT nbits is actually bits_per_sample / ndim. For the scan config, the nbit relates to the input bit-width, but for the DADA header and DPD, this field should be the output bit width. For VR, input==output, but for the other modes these will be different.
oversampling_ratio OS_FACTOR Config us a tuple of 2 values, but
frequency_band UDP_FORMAT There is a mapping from frequency band to UDP format: "low" -> "PstLow", for mid the "5a" and "5b" map to "MidPSTBand5" for bands 1 to 4 the value is "MidPSTBandX", where X is the band number
timing_beam_id BEAM_ID If timing_beam_id is not set it is the BEAM id of the TANGO device
activation_time ACTIVATION_TIME
observer_id OBSERVER context/observer
project_id PROJID
pointing_id PNT_ID
subarray_id SUBARRAY_ID
source SOURCE obscore/target_name
itrf ITRF The input is an array but the header value is a string joined by a ","
coordinates/equinox EQUINOX If the equinox is not set our config defaults to 2000.0
coordinates/ra STT_CRD1 obscore/s_ra This is in hour angle
coordinates/dec STT_CRD2 obscore/s_dec This is in deg
max_scan_length SCANLEN_MAX max_scan_length is a float so we do an int() conversion
test_vector_id TEST_VECTOR
total_bandwidth BW total_bandwidth is in Hz and bandwidth is in MHz
num_frequency_channels NCHAN obscore/em_xel
centre_frequency FREQ Config is in MHz, output is in MHz
bandwidth_out BW_OUT This parameter could differ between input/output for FT/PT/DS
ft/num_bits_out NBIT_OUT
ft/polarizations POLN_FT
ft/channels CHAN_FT The config has this a tuple/array that is joined with a ","
ft/requantisation_scale DIGITIZER_SCALE
ft/requantisation_init_time DIGITIZER_INIT_TIME

In the above table, if the config key starts with ft/ then it’s specific for flow through mode. All other values are valid for all modes. As more modes are added the prefixes of ds/ and pt/ will be added too specific for the dynamic spectrum and pulsar timing modes respectively.

Fixed/Calculated mapping values

The following are calculated or set to a fixed value by PST. An example is that LMC calculates the subband resources which the *X*_OUT values are determined, even if at the moment we only use 1 subband. Similarly values like TSAMP, RESOLUTION are calculated from other values based on the scan configuration.

DADA Header Key Subband? Value Notes
NSUBBAND N 1 PST only uses 1 subband atm, however, in the future this will depend on amount of data throughput
BMAJ N 0.0 There is currently no mapping. Not available in scan configuration
BMIN N 0.0 There is currently no mapping. Not available in scan configuration
COORD_MD N "J2000" We only support J2000
TRK_MODE N "TRACK" Only support tracking mode of "TRACK"
START_CHANNEL Y 0 This is per subband, but as we only have 1 subband this starts as 0
END_CHANNEL Y NCHAN This is per subbabd, but as we only have 1 subband this is nchan - use exclusive range
START_CHANNEL_OUT Y 0 This is per subband, but as we only have 1 subband this starts as 0
END_CHANNEL_OUT Y NCHAN This is per subbabd, but as we only have 1 subband this is nchan - use exclusive range
NCHAN_OUT Y NCHAN The number of channels out, currently set to nchan
FREQ_OUT Y FREQ Config is in MHz, output is in MHz
NDIM N 1 - for weights
2 - for data		 PST only supports complex valued data but the weights file is real valued
TSAMP N 1 / (BW / NCHAN * OS_FACTOR)
BYTES_PER_SECOND N NCHAN * NPOL * NBIT * 2 / 8e6 / TSAMP This is specific to the observation mode, for VR this calculation is correct but it should be NCHAN_OUT, NPOL_OUT and NBIT_OUT
DATA_HOST N specific to env This is specific to RECV.CORE and the host IP it is on
DATA_PORT N specific to env This is specific to RECV.CORE and port number to receive data on
TELESCOPE N "SKALow" or "SKAMid" This depends on which telescope and the UDP format used
UTC_START N The start time in UTC of the scan to the precision of a second.
PICOSECONDS N The fractional time in picoseconds after the UTC_START that the scan actually started
RESOLUTION N (NSAMP_PER_PACKET * NCHAN * NDIM * NPOL * NBIT) / 8 The number of bytes to have samples for all the channels. Note that this value in a header file is calcuated differently.
OBS_OFFSET N The data offset from the start of a scan that the file is for, this should be a multiple of the RESOLUTION closest at or beyond 10 seconds
FILE_NUMBER N The file number, starting from 0
NANT N len(ANTENNAE) This is equal to len(receptors)

DPD Metadata fields

The following fields don’t have a direct mapping back to the scan configuration

Key Value / Default CSP Sources Notes
execution_block EB_ID execution_block_id
context/intent "Tied-array beam observation of " source
context/notes "Unknown" N/A If INTENT is in the headers we could get this
config/image "artefact.skao.int/ska-pst/ska-pst" N/A This is the CONFIG_IMAGE constant within SEND code
config/version "0.1.3" N/A This is the CONFIG_IMAGE constant within SEND code - should check if this should just be the version of the application
obscore/dataproduct_type "timeseries" N/A When sending different observation modes we will need to update this
obscore/dataproduct_subtype "voltages" N/A When sending different observation modes we will need to update this
obscore/calib_level 0 N/A
obscore/obs_id SCAN_ID scan_id
obscore/access_estsize sum of just the data files data sections N/A
obscore/s_ra STT_CRD1 coordinates/ra This is in hour angle
obscore/s_dec STT_CRD2 coordinates/dec This is in deg
obscore/t_min UTC_START N/A value is in MJD calculated from string. Note the PICOSECONDS is missing.
obscore/t_max t_min + TSAMP N/A not sure if this is correct or if t_max should be t_min + scan_length that can be calculated all the files
obscore/t_resolution RESOLUTION frequency_band, nchan, num_of_polarizations, and bits_per_sample This probably be TSAMP as RESOLUTION is a size in bytes not seconds
obscore/t_exptime TSAMP total_bandwidth, nchan and oversampling_ratio this is the exposure time, should be length of scan - potential bug here. What would the value be for Pulsar Timing observation mode?
obscore/facility_name "SKA-Observatory" N/A
obscore/instrument_name TELESCOPE frequency_band Frequency band can be used to work out value. DPD puts a - after the SKA (i.e. SKA-Low or SKA-Mid)
obscore/pol_xel NPOL num_of_polarizations
obscore/pol_states "null" N/A
obscore/em_xel NCHAN nchan
obscore/em_unit "Hz" N/A
obscore/em_min (FREQ - BW/2) * 1e6 centre_frequency and total_bandwidth Units in Hz not MHz so there is a factor of 1e6
obscore/em_max (FREQ + BW/2) * 1e6 centre_frequency and total_bandwidth Units in Hz not MHz so there is a factor of 1e6
obscore/em_res_power "null" N/A
obscore/em_resolution (BW / NCHAN) * 1e6 centre_frequency and nchan Units in Hz not MHz so there is a factor of 1e6
obscore/o_ucd "null" N/A

Potential bugs in mapping

  • config/version be the version of ska-pst that sent it and not be hardcode to 0.1.3?

  • obscore/dataproduct_type and obscore/dataproduct_subtype are currently hardcoded for VR mode

  • obscore/t_min is not including the fractional seconds of the start time

  • obscore/t_max should be t_min + scan_length

  • obscore/t_resolution is currently RESOLUTION but should be TSAMP

  • obscore/t_exptime is currently TSAMP but should be scan_length

Config values currently not mapped

The following keys are on the scan configuration that are not sent to RECV or DSP which means they don’t get in to the output files

  • num_rfi_frequency_masks - this should map to NMASK. However, there PST needs to review how RFI masking is performed

  • rfi_frequency_masks - this should map to FREQ_MASK. However, there PST needs to review how RFI masking is performed

  • destination_address - this should be where to send data to for SDP. Currently we use a volumne mount

  • num_channelization_stages - this is the number of channelisation stages to do and should map to NSTAGE.

  • channelization_stages - this is a list of values that should map to NSTAP_k, COEFF_k, NCHAN_PFB_k, and OVERSAMP_k

  • subint_duration - this should map to OUTSUBINT

Documentation mismatch

  • source - says should map to SRC_NAME but we use SOURCE