Measurement Sets

Measurement Sets are the main data product output by the vis-receive pipeline.

Creation

The vis-receive pipeline creates Measurement Sets in the Measurement Set version 2 format (see the documentation about supported standards and MS creation in the supporting Python package), with minor exceptions noted below.

For any given observation, individual Measurement Sets are written per beam, scan and reception node under the following directory structure in the data product PVC:

/product/<eb-id>/ska-sdp/<pb-id>/<beam-id>/<scan-id>/output.<scan-type-id>.<subband>.ms

where:

  • eb-id is the Execution Block ID.

  • pb-id is the Processing Block ID.

  • beam-id is the (visibility) beam ID.

  • scan-id is the scan ID.

  • scan-type-id is the scan type ID.

  • subband is the vis-receive node rank (starting at 0).

Measurement Sets are not written for scan-0 data.

Contents

This section describes how most of the CBF visibility data and corresponding metadata flows into the output Measurement Set. For full details on how tables are set up and written in Measurement Sets, visit the documentation for each table in the supporting Python package.

Metadata

Metadata stored in Measurement Sets is provided at creation time (see MS creation). Much of it originates from the AssignResources command sent to SDP:

  • A list of antennas is built by augmenting and sorting the list of receptors specified in the resources.receptors values in the AssignResources command. Receptors explains how this process works in detail. Its information is written to the ANTENNA subtable (see ANTENNA table).

  • The scan information includes the scan intents and integration time. The former is stored in the STATE subtable (see STATE table), the latter is mixed in several ways together with incoming data (see CBF correlated data).

  • The beam information is sourced from one of the beams of the scan type. The beam is identified by matching the numeric beam ID sent by CBF against the visibility_beam_id in the SDP beam definition. In SKA Mid, the CBF is allowed not to send a beam ID, in which case a single beam with type visibilities must be defined in SDP. Its associated field information is stored under the FIELD table (see FIELD table). Its associated polarisation information is stored under the POLARIZATION table (see POLARIZATION table).

  • The spectral window corresponds to the subset of frequencies received in a given node. At the moment of writing, vis-receive supports neither multi-node reception nor zoom windows; therefore, this object is equal to the full beam’s spectral window. Its information is written to the SPECTRAL_WINDOW subtable (see SPECTRAL_WINDOW table).

  • Baseline information defaults to a lower-triangular set of baselines, matching the baseline order used by CBF. Setting the mswriter.upper_triangular_baselines parameter to True causes the data to be re-ordered so it is written in upper-triangular baseline order instead.

  • The following observation metadata is provided, all written to the OBSERVATION subtable (see OBSERVATION table).

    • The telescope name is set to either (in order of priority):

    • The Scan ID.

    • The Processing Block ID.

    • The Execution Block ID associated with the Processing Block executing the vis-receive pipeline.

  • The Measurement Set version is set to MSv2.

CBF correlated data

For a given interval value (provided to SDP for each scan type via its AssignResources command), the following values are written to the Measurement Set’s main table for the stream of data coming out of CBF (see MAIN table).

  • Times written to the TIME column are calculated as cbf_timestamp - interval / 2.

  • Visibility data is written to the DATA column. Other than swapping the baseline and frequency dimensions, and optionally changing the baseline ordering when mswriter.telescope_name is set to True, data is stored unmodified.

  • Exposure values are calculated as interval * (FD / 255) ^ 2. Because they are sent per channel by CBF, they are averaged before writing to EXPOSURE.

  • Weights are set to exposure * bandwidth_in_khz using the exposure values calculated from CBF’s FD, resulting in per-frequency weights. These are therefore written to WEIGHT_SPECTRUM, and their average across frequencies to WEIGHT.

  • Time centroids are calculated as time + interval * TCI / 256.

Flagging

Data is automatically flagged before being written to the Measurement Set under the following conditions:

  • If visibility data contains NaN, Inf or -Inf, it is flagged.

  • If the time centroids for a given interval and baseline differ by more than 10% across frequency channels, the affected data is flagged.

  • If visibility data is received from CBF for certain channels but not others, the missing-channels data is flagged.

    Note

    If visibility data is missing for all frequency channels at a given timestamp, that timestamp is omitted entirely rather than being written with flags.

Receptors

The vis-receive pipeline is required to sort the resources.receptors values in the AssignResources command sent to SDP according to the sorting rules of each of the Low and Mid CBF SDP ICDs. This is done as follows:

  • First, details about the telescope and its receptors are loaded from the telescope layout document of the telescope model, pointed to through the telescope_model.layout_key and telescope_model.sources parameters. Details include, among other things, human-readable labels, receptor IDs, and location.

  • Receptor names provided in AssignResources are then identified as belonging to either the SKA Low or SKA Mid telescope:

    • If all receptor names follow the name pattern SKAnnn or MKTnnn, SKA Mid rules are assumed, where receptors need to be sorted by their DishID. This is equivalent to sorting them lexicographically in their string form (see ADR-32).

    • Otherwise, SKA Low rules are assumed. The sorting rule is that receptors need to be sorted by station ID first, then by substation ID.

In the case of SKA Low, receptors can be given either as aperture identifiers or as station labels:

  • If all receptor names follow the pattern APxxx.yy they are interpreted as aperture identifiers (see ADR-62), where xxx is the station ID, and yy is the substation ID.

  • Otherwise, receptor names are assumed to be station labels, and are required to all match receptor labels loaded from the telescope layout document, where their station ID is loaded from. Their substation ID is assumed to be 0.

Additionally, when receptors are provided as aperture identifiers, a user-friendly receptor label is assigned to them:

  • If the substation ID is 0, the full station is assumed, and therefore the label loaded from the telescope layout document is used as-is (e.g., s8-1).

  • If the substation ID is any other value, the resulting label has the form <label_from_layout_document>.<substation_id:02d> (e.g., s8-1.03).

Pointing information

The vis-receive pipeline can be configured to capture pointing information from the SKA Mid and SKA Low telescopes. This information is then stored in the POINTING table (see POINTING table).

Pointing information is captured by providing the appropriate telstate parameter to the script. This parameter consists of a number of per-beam, per receptor Tango Resource Locators (TRLs) for device attributes that are subscribed to for obtaining pointing information.

Note

As of version 9.0.2, additional command line parameters still need to be provided to the mswriter processor component of the vis-receive pipeline. This will not be required in version 10.0.0 onwards, so this existing manual intervention is not described here.

Depending on the set of TRLs that are provided, different behaviour takes place:

  • If target_trls and tracking_trls are provided, a “tracking consumer” logic takes place. Under this scheme, any update on a tracking attribute results in a new row in the POINTING table, where the TRACKING column is set to the new value, and the TARGET and DIRECTION columns are set to the latest value observed from the corresponding target attribute.

  • If target_trls and direction_trls are provided (and optionally source_offset_trls), an “interpolating consumer” logic takes place instead. Under this scheme, all attribute updates are first accumulated in memory as-is. Once data needs to be written to disk, target attribute values are first interpolated to match the timestamps of the direction values. If source offset attributes were subscribed to, their values are interpolated in the same way, otherwise they are calculated as the difference between the direction and interpolated target attribute values.

Pointing data is written to disk at the end of a scan, when the Measurement Set is being closed.

Phased Array information

In the case of the SKA Low telescope, information about the antenna elements of a station is stored in the output Measurement Sets under the PHASED_ARRAY subtable.

Note

The supporting Python package used for creating and writing data to Measurement Sets does not include the creation and population of the PHASED_ARRAY table, which is why it is described here instead. This might change however in the future, at which point the PHASED_ARRAY table description will be migrated to live with the rest of the Measurement Set table descriptions.

Data for each station is loaded from <station_data_key_path>/<station_label>.yaml document in the telescope model data, where:

Within each document, the station information is rooted at platform.stations.<station_label>. The schemas for the station documents can be found here.

Phased array information is stored under the PHASED_ARRAY table (see PHASED_ARRAY: phased array station information). However, a slightly different layout compatible with older versions of the Everybeam and OSKAR packages is used, where the ELEMENT_FLAG column dimensions are swapped, and the number of polarisations is fixed at 2. This resulting structure is similar to the LOFAR_ANTENNA_FIELD table (see MeasurementSet description for LOFAR, Version 2.08.01, page 22). Everybeam version 0.8.2 and OSKAR version 2.13.0 both added support for the standard layout; therefore future versions of the pipeline will use the standard PHASED_ARRAY table structure from the Measurement Set definition version 3.0𝛽.

The PHASED_ARRAY table is structured as follows, where \(N_{ant}\) is the number of antenna elements in the station, one row per station.

Column

Data Type

Units

Measure

Value

Source

ANTENNA_ID

int

Number of receptor as per ANTENNA table

Sequential number of receptor after ICD reordering, see Receptors.

COORDINATE_AXES

double(3, 3)

direction

Rotation matrix from ITRF to station ENU reference frame

Computation using reference and rotation from station information

ELEMENT_FLAG

bool(2, \(N_{ant}\))

True if antenna element is masked, False otherwise

Corresponding antennas[*].masked values

ELEMENT_OFFSET

double(3, \(N_{ant}\))

[‘m’, ‘m’, ‘m’]

position, ref: ITRF

Rotated antenna offsets

Computation using COORDINATE_AXES and antennas[*].location_offset

PHASED_ARRAY_ID

int

Same as ANTENNA_ID

POSITION

double(3)

[‘m’, ‘m’, ‘m’]

position, ref: ITRF

Station position

Computation using reference and rotation from the station information