Running the pipeline in SDP
The pointing offset calibration pipeline can be run either as a stand-alone pipeline or as an SDP processing script alongside the vis-receive script. This page is concerned with the latter. Refer to the documentation of the processing script to see how you can execute it within SDP:
https://gitlab.com/ska-telescope/sdp/ska-sdp-script/-/blob/master/src/pointing-offset/README.md
Pointing data sent to a Kafka queue
The pointing pipeline communicates its results with the rest of the telescope system via a Kafka queue. It publishes the following values as a single, structured numpy array encoded with msgpack_numpy:
DATA_STRUCTURE = numpy.dtype(
[
("antenna_name", "U8"),
("last_scan_index", "f8"),
("xel_offset", "f8"),
("xel_offset_std", "f8"),
("el_offset", "f8"),
("el_offset_std", "f8"),
("expected_width_h", "f8"),
("expected_width_v", "f8"),
("fitted_width_h", "f8"),
("fitted_width_h_std", "f8"),
("fitted_width_v", "f8"),
("fitted_width_v_std", "f8"),
("fitted_height", "f8"),
("fitted_height_std", "f8"),
]
)
The Data Product page describes what these values mean.
xel
stands for cross-elevation and el
for elevation.
v
stands for vertical, h
for horizontal co-polarisation, while
std
is the standard deviation of the given value.
The default Kafka topic where the results are published to is pointing_offset
.
This can be customized using the SDP_KAFKA_TOPIC
environment variable.
Connecting to the Configuration Database
The Configuration Database is the centre information store of SDP. The SDP components connect to it and navigate, perform their tasks, based on the information stored there.
The pointing offset pipeline takes scan information from the execution block state.
Once it is deployed, it waits for the first pointing
scan type to start (from the last processed scan ID).
Next, it collects the IDs of the scans that have the type pointing
and status FINISHED
that have not yet been processed.
We need to tell it how many scans we expect in a batch so that it knows how long to wait for.
The scan ID information is needed to determine which MeasurementSets to read into the
pipeline for further processing, once all the relevant scans are finished.
The script can receive and process multiple batches of scans. Once the pointing script has been deployed, the processing block that executes the pipeline will be kept running and automatically identify the last processed scan ID and process the batch as they are sent.
If an exception occurs during processing the error information will be stored in the processing block state for that batch of scans.
The entry point for the pipeline is the pointing_offset
command, which runs the
pointing_offset_cli. In order to connect to the Configuration DB and perform the tasks described above,
the SDP_PROCESSING_SCRIPT
environment variable has to be set to True
.
This run-mode requires an execution block ID and an integer telling the pipeline how many
pointing scans we expect to happen before processing.
Capturing errors
Sometimes the pipeline will output an error. The current design
of the pipeline captures if a batch of pointing scans are unable to deliver a valid
array of offsets, and stores the erroneous scan IDs as well as the error messages in
the processing block state, under the processed
entry. This is a list of processed scans,
their status (success or fail), and if they failed, what the error message was.