SDP Data Product Metadata

SDP Data Product Metadata is a Python package to record SKA-specific metadata alongside the data products. It creates metadata files containing:

• Execution block ID

• The context of the execution block provided by the Observation Execution Tool (OET)

• The configuration of the processing block used to generate the data products

• A list of data product files with description, status, size and a cyclic redundancy check (CRC) for error detection

• Associated IVOA ObsCore attributes used for querying astronomical observations

• Contents of Metadata file
• API

Metadata file location and directory structure

The metadata path is constructed dynamically from the Execution Block ID, Processing Block (PB) ID, and optional subpath components (for example, beam ID or scan ID). These optional components allow the metadata file to be placed in the required directory hierarchy.

In the example below, the ObsCore section of ska-data-product.yaml is populated with the observation metadata from measurement_set.ms located in the same directory, with beam_id and scan_id provided as subpath components. If no optional subpath components are supplied, the metadata file is written directly under the PB directory.

Example directory layout:

/product/<eb-id>/ska-sdp/<pb-id>/
    └── <beam_id>/
        └── <scan_id>/
            ├── measurement_set.ms
            └── ska-data-product.yaml

Indices and tables

• Index