Pointing offset script

This script deploys the SKA pointing offset pipeline, which calculates pointing offsets from “pointing”-type scan data.

Deploying in SDP

To set up SDP on your local machine using Minikube (or a remote cluster), follow the instructions at Installing SDP. If you want to test the script, we suggest you deploy SDP together with itango.

Configuring the Script

The pointing-offset script can be configured through the following Processing Block parameters:

Name	Description	Default
`image`	OCI image of the pointing offset pipeline	`artefact.skao.int/ska-sdp-wflow-pointing-offset`
`version`	Version of the OCI image of the pipeline	`0.4.0`
`num_scans`	Expected number of scans in a pointing observation	`5`
`kafka_topic`	Kafka topic to send pointing data offset results to	`pointing_offset`
`encoding`	Encoding method used to encode/decode the offsets	`msgpack_numpy`
`additional_args`	Additional arguments to be passed to the pointing-offset pipeline command	`[]`
`env`	User-defined environment variables	`[]`
`telescope_model`	Dictionary of path and file values to specify static data for the pipeline	`{}`
`tango_attribute`	Prefix to the tango attribute name which will hold the pointing offsets on the QueueConnector device (i.e. “_{dish_id}”)	`pointing_offset`

The pointing-offset helm chart requires an args and an envs values, which the processing script itself updates directly. args default to running the following command:

pointing-offset compute --msdir <ms_dir> --eb_id <eb_id> --num_scans <num_scans>

where ms_dir is the directory where the Measurement Sets (MS) are found and is constructed as follows:

ms_dir = "/mnt/data" + f"/product/{eb_id}/ska-sdp/{pb_id}/"

and eb_id is the execution block ID, pb_id is the processing block ID, and num_scans is the expected number of pointing scans with MS data to be processed, defined by the num_scans processing block parameter.

Note: if there is a vis-receive script running, then the pb_id needs to match that of the vis-receive script’s PB, since the MS data will be written into that script’s associated directory. If no vis-receive script is running, pb_id is set to the pointing script’s ID. The pb_id of the vis-receive script is passed to the pointing-offset script via the dependencies argument of the processing block.

Additional command line arguments can be provided by setting the additional_args parameter in the processing block parameters. This has to be a list, e.g.: ["--num_chunks", "1"]. These will be added to the pointing-offset command.

You may specify the data you want the pointing pipeline to use from the Telescope Model Data Repository by specifying the sources and file path needed. Currently only loading static RFI masks is supported, which can be set up as the following example shows:

"parameters": {
    "telescope_model": {
        "telmodel_sources": ["gitlab://gitlab.com/ska-telescope/ska-telmodel-data?yan-1618-add-rfi-model#tmdata"],
        "static_rfi_key": "instrument/ska1_mid/static-rfi/rfi_mask.h5"
    }
}

In addition, set the "--apply_mask" additional argument to actually use the mask.

The following environment variables are added to the env parameter:

Name	Description	Default
`SDP_PB_ID`	Processing block ID (get from SDP)	None
`SDP_PROCESSING_SCRIPT`	Tells the pointing pipeline that it is run as an SDP script	`True`
`SDP_CONFIG_HOST`	Host address of the Configuration DB	`127.0.0.1`
`SDP_CONFIG_PORT`	Port of the Configuration DB	`2379`
`SDP_KAFKA_TOPIC`	Kafka topic to send pointing offsets to	`pointing_offset`
`SDP_KAFKA_SERVER`	Kafka server (host)	`localhost:9092`
`SKA_TELMODEL_SOURCES`	A list of telescope model sources as provided by the `telmodel_sources` parameter	`[]`

SDP_PROCESSING_SCRIPT always needs to be “True”, else the pointing pipeline will not check the Configuration DB for scan information, instead it will run in a stand-alone mode, assuming the user provides input data manually.

SDP_CONFIG_HOST and SDP_CONFIG_PORT are needed for establishing a connection to the Configuration DB and is specified directly by SDP, when the script is run inside SDP.

SDP_KAFKA_TOPIC is updated using the kafka_topic processing block parameter if it exists, or defaults to the values in the table. SDP_KAFKA_SERVER is based on the KAFKA_HOST environment variable passed to the script by the SDP Processing Controller at runtime.

You may specify additional environment variables in the script parameters, which will be merged with the above ones. E.g. you may want the pointing_offset results to be displayed in a JSON format on the QueueConnector tango device. For this, set the following ProcessingBlock parameter:

"env": [
    {
      "name": "USE_POINTING_TABLE",
      "value: "True"
    }
]

Additionally, if you want the format in JSON, set the encoding argument to json.

The script also sets the dataProductStorage name argument, loading it from the SDP_DATA_PVC_NAME, which is passed to it by the ProcessingController, and it points to the PVC deployed by SDP upon start.

Script Helm Chart

The execution engine for the pointing offset calibration pipeline is the pointing-offset chart. For more customizable chart variables, follow the link to the documentation.

Configuring receive addresses

The pointing-offset script is expected to provide the pointing offset Fully Qualified Domain Name (FQDN) in the receiveAddresses tango attribute of the SDP subarray. The script itself writes the information to its processing block state, and the subarray device takes this information and combines it with the host and port values generated by the vis-receive script, which is then stored in the tango attribute.

The pointing FQDN is stored in the pointing_cal key defined by schema 0.5 (and above) of receive addresses. It is available for every scan_type but only for “visibility”-type beams.

Step-by-step testing in SDP

This section gives an example of how one can run the pointing processing script, together with the vis-receive script, in SDP. The example uses the Mock Dish devices to send pointing data and the CBF emulator to send visibility data.

1. Set up the sender and receive data pods

Both of these pods mount a Persistent Volume Claim (PVC) to hold data. The sender-data pod is needed to provide the input data for the Mock Dishes and the CBF emulator. The receive-data pod is needed to inspect the outputs of the receiver and the pointing pipeline.

The testing suite of SDP provides pod definitions for both:

The sender-data pod has to be deployed in the SDP control-system namespace, while the other needs to be deployed in the SDP processing namespace. Make sure you update the PVC claim name to the PVC used by SDP in the above files (usually test-pvc):

volumes:
- name: data
  persistentVolumeClaim:
    claimName: <sdp-pvc-name>

To install them, run these kubectl commands (use the appropriate file and namespace):

kubectl create -f <sender-pod-yaml> -n <control-system-namespace>
kubectl craete -f <receive-pod-yaml> -n <processing-namespace>

2. Download the data for testing

The MeasurementSets (MS) used for testing the pointing pipeline can be found on GCP. Use the outer scans 1, 4, 5, 8, 9 for a 5-point test. If you need to run a 9-point test using both inner and outer scans, refer to the “split_into_scans” folder instead.

The pointing HDF files on GCP can be found here. There is a tempered_data directory containing HDF files for the 5 scans, which have been modified to contain data that the MockDishes expect. Unfortunately, the data aren’t correct, but its structure is needed to be able to run our test. Note that this results in incorrect pointing offset data produced by the pipeline. Use the data in this directory. Eventually, we will replace the test data with correct ones.

Download the data in to the sender-data pod. Exec into the pod using kubectl or k9s (link here)

kubectl exec -it sender-data -n <control-system-namespace> -- bash

Install gsutil

pip install gsutil

and download the MS data and pointing data from the GCP link above into the mounted PVC (usually /mnt/data/), later the CBF emulator needs to be pointed there.

3. Install the Mock Dish helm chart

The chart can be found in the SDP integration repository:

mock-dish

Inspect tha values.yaml file, add the data path for each scan you are planning to run, for both the dishMaster (achievedPointing) and dishLeafNode (desiredPointing) devices. For now, replace the entries of sourceOffset on the dishLeafNode with {}, since we do not use that data in this set up. Update the number of data_antennas: the example MS files contain data from four antennas. Also update the pvc.name value with the PersistentVolumeClaim that contains the data. Finally, add the correct list of receptors to the receptors value. This needs to match what is later added to the AssignResources configuration string.

To install, run:

helm install mock-dish ska-sdp-integration/tests/resources/charts/mock-dish

(update path if needed, to where you have the chart).

4. Run AssignResources on the subarray device

Exec into the itango pod (with kubectl or k9s)

kubectl exec -it ska-tango-base-itango-console -n <control-system-namespace> -- itango3

This only needs to be done once.

Connect to the device:

d = DeviceProxy('test-sdp/subarray/01')
d.On()

Load the configuration string into a variable:

import json
config = json.dumps(<example_configuration_string>)

The configuration string used for this test can be found here.

Note the following about this string and update as instructed:

it runs two processing scripts in two different processing blocks (PB):
- the pointing-offset script runs with PB ID pb-test-20211111-00000
- the vis-receive script runs with PB ID pb-test-20211111-00001
the pb_id in the pointing-offset script’s dependencies key needs to match the pb_id of the vis-receive script
similarly, the pb_id in the vis-receive script’s dependencies key needs to contain the pb_id of the pointing-offset script. If this is omitted, the QueueConnector device will not load pointing data from the dish tango devices.
(if you rerun the script, either delete the PB information from the configuration DB, or update the IDs in the configuration string)
resources.receptors refer to SKA Mid dish names, these are random, at the moment the important thing is the number of them, not their actual names.

Run AssignResources:

d.assignResources(config)

This will start the two processing blocks in the processing namespace, which, in return, will start the pod for the pointing pipeline and the pod for the receiver, respectively.

Once the obsState reached IDLE, run Configure for pointing scans:

d.Configure(json.dumps({
  "interface": "https://schema.skao.int/ska-sdp-configure/0.4",
  "scan_type": "pointing"
}))

5. Running Scans

From this point, the below commands need to be run for every scan. Once the obsState of the subarray device is READY (you can check by printing d.ObsState), turn on the dishes (both Dish LeafNodes and DishMaster devices), so that the pointing data can be accessed from their attributes:

devs = [dev for dev in get_device_list() if 'mockdish' in dev]

for dev in devs:
    dish = DeviceProxy(dev)
    dish.Scan(<scan_id>)

Next, run the first scan:

d.Scan(json.dumps({
  "interface": "https://schema.skao.int/ska-sdp-scan/0.4",
  "scan_id": <scan_id>
}))

Replace <scan_id> with the current scan ID (integer).

Set up the CBF emulator to send the first MS (in the command line, not in tango). The helm chart can be found in the SDP integration repository:

cbf-sender

Inspect and update the values.yaml file, including the name of the pvc (pvc.name). Add a top-level parameter for the command attribute:

command:
  - "emu-send"
  - "/mnt/data/1672727931_ordered_scan_1.ms"
  - "-o"
  - "transmission.transport_protocol=tcp"
  - "-o"
  - "transmission.method=spead2_transmitters"
  - "-o"
  - "transmission.num_streams=1"
  - "-o"
  - "transmission.rate=10416667"
  - "-o"
  - "reader.num_repeats=1"
  - "-o"
  - "transmission.target_host=proc-pb-test-20211111-00001-vis-receive-00-0.proc-pb-test-20211111-00001-vis-receive.<processing-namespace>"
  - "-o"
  - "transmission.target_port_start=21000"
  - "-o"
  - "transmission.scan_id=<scan_id>"
  - "-o"
  - "transmission.telescope=mid"

transmission.target_host: update the namespace in this string. If you changed the processing block ID for the vis-receive, then update the relevant part of the string with the correct ID.

For each scan, modify the transmission.scan_id for the right scan number and "/mnt/data/1672727931_ordered_scan_{scan_id}.ms", which needs to point to the file you are sending, in the sender-data pod.

Make sure the pvc name in the values.yaml also matches with the pvc on your machine.

Install the chart:

helm install cbf-send-1 ska-sdp-integration/tests/resources/charts/cbf-sender

(change the path to the chart as appropriate for your setup)

Once the emulator finished running (the pod status shown as completed), back in itango, end the scan:

d.endScan()

Turn off the mock dishes:

for dev in devs:
    dish = DeviceProxy(dev)
    dish.EndScan()

6. Repeat for all scans

Repeat four more times steps 5:

Turn on the mock dishes, using the next scan_id
Run d.Scan with the right scan_id
Install the CBF emulator with an updated command: scan_id and MS name
End the scan and turn off the dishes

The pointing offset pipeline is triggered when the fifth endScan command is executed. It will read the MS data sets that can now be accessed in the receive-data pod, run its processing, then send the output offsets to an attribute of the QueueConnector Device. You can monitor the progress of the pipeline by looking at the logs of the pod in the processing namespace, e.g. pod name proc-pb-test-20211111-00000-pointing-offset*.

7. Retrieving data from the QueueConnector Device

The SDP QueueConnector device needs to be configured to read data from a kafka topic and save it to an attribute. The pointing-offset script automatically configures this, with defaults to msgpack-numpy encoding and pointing_offset tango attribute prefix.

The actual attribute name will contain the dish ID, e.g. pointing_offset_SKA001. There is one attribute per dish. Each attribute contains spectrum data of three floating point values: [scan_id, xel_offset, el_offset]. The scan_id is the ID of the last scan of the given pointing observation. Cross-elevation (xel) and elevation (el) offsets are given in degrees.

After the pipeline run is complete, in itango3, run the following code to display the data (replace the dish ID as requred):

q = DeviceProxy("test-sdp/queueconnector/01")
q.pointing_offset_SKA001

8. Finishing and clean up

Once the pointing pipeline finished, it will wait for the next set of pointing scans. To remove the deployment, you need to end the observation by returning the subarray to an EMPTY state (in itango):

d.end()
d.releaseAllResources()

If you want to remove all the resources you deployed, uninstall the following helm charts (run in the command line): five cbf-send-<id> charts, mock dish chart, SDP chart:

helm uninstall cbf-send-1  # repeat for 2, 3, 4, 5
helm uninstall mock-dish
make uninstall-sdp  # if you used the make targets to install

To remove the sender and receive pods, run:

kubectl delete -f <sender-pod-yaml> -n <control-system-namespace>
kubectl delete -f <receive-pod-yaml> -n <processing-namespace>

Additional: Capturing errors

The steps above assume the pointing pipeline executes successfully, output valid offset values for each antenna. However, sometimes the pipeline will output an error. The current design of the pipeline captures if a batch of pointing scans are unable to deliver a valid array of offsets, and store the erroneous scan IDs as well as the error messages in the processing block state. You can review the error messages in the SDP console (For full documentation of the ska-sdp CLI, see SDP command line interface ):

ska-sdp get /pb/{PB_ID}/state

Under the processed entry there will be a list of processed scans, their status (success or fail), and if they failed, what the error message is.

Changelog

0.5.0

Minimum requirements

Pointing offset calibration pipeline (ska-sdp-wflow-pointing-offset): 0.5.0
SDP (ska-sdp-integration): 0.19.0
Queue Connector (ska-sdp-lmc-queue-connector): 4.1.0 Note: only SDP 0.21.0+ runs version 4.1.0 of the Queue Connector by default; previous versions will need to manually set it
vis-receive: 4.1.0

Changes

Documentation updates (MR129)
Allow customization of the pointing offset tango attribute and update receiveAddresses (MR128)
Update the QueueConnector exchange for pointing offsets to provide data on an attribute per dish (MR125, MR129)

0.4.0

Set minimum suggested pipeline version to 0.4.0 (MR121)
Load telescope model data path and file information from script parameters (MR118, MR120)
Update documentation (MR121, MR116, MR115)
Refactored the pointing script into functions and added unit tests (MR112)
Delete unused arguments and set the results directory (MR113, MR111)

0.3.1

Use ska-sdp-scripting 0.6.3, which implements configuring the QueueConnector device in Phase enter and exit methods (MR106)

0.3.0

Update example JSON string and documentation (MR98, MR99, MR101, MR100)
Update receive addresses with pointing FQDN (MR95)
Use script dependencies to obtain pb_id of the vis-receive script (MR94)
Script no longer configures the QueueConnector device directly (done via vis-receive)(MR87)

0.2.0

Update documentation to include usage of mock dishes and CBF emulator (MR83)
Update to scripting library v0.6.1 (MR84)

0.1.0

Configure the QueueConnector device to store pointing offsets and send Kafka information to pipeline via environment variables (MR82)
Load additional arguments, if any, to CLI args from parameters (MR81)
Load PVC information from environment variable (MR81)

0.0.2

Use the vis-receive script’s PB ID (if exists) for determining the MS directory (MR80)

0.0.1

Initial version of the pointing script (MR74)