Components

../_images/sdp_components.svg

SDP components and the connections between them.

The diagram shows the components of the SDP system and the connections between them. The components, grouped by their function, are as follows.

Execution Control:

  • The Configuration Database is the central store of configuration information in the SDP. It is the means by which the components communicate with each other and it is implemented using etcd.

  • The Controller Tango device is intended to provide the top-level control of SDP services. Presently, it performs the state transitions in response to commands, reports the status and versions of the SDP system and its components and updates the healthState based on components’ status.

  • The Subarray Tango devices control the processing associated with the SKA subarrays. They are the primary interface between the SKA software ecosystem and the SDP sub-system.

  • The Queue Connector Tango devices provide the interface for exchanging real-time data with other telescope subsystems over Tango. Within the SDP, the data travel over the data queues to and from processing deployments.

  • The Console provides a command-line interface to the configuration database to control and monitor the system. This is mainly a developer interface.

  • The Processing Controller controls the execution of processing blocks. It detects them by monitoring the configuration database. To execute a processing block, it requests the deployment of the corresponding script by creating an entry in the configuration database.

Quality Assessment (QA):

  • The Display Server serves the client with the HTML and JavaScript required to display QA data provided by the API server.

  • The API Server reads QA data from the data queues and serves them to the client to display. Data for the current setup and metrics are available via WebSocket connections, Information regarding the subarrays, processing blocks, and current setup is also available in REST endpoints.

Data Queues:

  • The Data Queues are the means by which components of the system exchange data in real time. They are implemented using Kafka.

Platform:

  • The Helm Deployer is the service that the platform uses to respond to deployment requests in the configuration database. It makes deployments by installing Helm charts (a collection of files that describe a set of Kubernetes resources) into a Kubernetes cluster.

  • Kubernetes is the underlying mechanism for making dynamic deployments to run realtime data processing and to deploy the SDP control system.

  • The Slurm Deployer allows batch jobs to be submitted to a Slurm cluster. It monitors the Configuration Database for deployment entries that request a Slurm script to be executed. When a Slurm deployment entry is added, the Slurm Deployer submits the corresponding job to the Slurm cluster via the REST API and monitors its state.

  • Slurm is the workload manager used for deploying and running batch processing pipelines.

  • The Buffer Manager currently determines the total storage space of the connected PVC for data processing.

  • The Global Sky model generates Local Sky Model from requests made by the Processing Blocks.

Processing Block Deployment:

  • A Processing Script orchestrates the execution of a processing block. It connects to the configuration database to retrieve the parameters of the processing block and to request the creation of processing deployments.

  • A Processing Deployment is created by the script to process data. It can read and write data products in the data product store. It may also exchange data in real time with other components of the system via the data queues, including producing data for the QA display. Specialised deployments are used to receive visibility data from the Correlator Beamformer.