Modules

SDP software modules.

The SDP is built from software modules which produce a number of different types of artefacts. The components of the system are built as Open Container Initiative (OCI) images which are deployed on a Kubernetes cluster using a Helm chart. The OCI images depend on libraries containing common code. The diagram shows the SDP modules and the dependencies between them.

The source code is hosted in the SKA Science Data Processor group in GitLab, in the following repositories:

• Integration

  Integration of components into the SDP system. Contains the Helm chart to deploy the SDP, the system integration tests, and this documentation.

• Configuration Library

  Library providing the interface to the configuration database.

• Etcd

  Etcd is the highly available key-value database underpinning the configuration library.

• Local Monitoring and Control

  Tango devices providing the local monitoring and control (LMC) of the SDP.

• LMC Queue Connector

  The Queue Connector Tango devices provide the interface for exchanging real-time data with other telescope subsystems.

• Processing Controller

  The Processing Controller controls the execution of processing blocks.

• Buffer Manager

  The Buffer Manager manages the buffer space, clearing up old data as needed.

• Helm Deployer

  The Helm Deployer provides the interface between the system and the Kubernetes platform.

• Slurm Deployer

  The Slurm Deployer sends slurm jobs to a Slurm cluster.

• Console

  Command-line interface to control and monitor the SDP.

• Processing Scripting Library

  Library providing the high-level interface for writing processing scripts.

• Processing Scripts

  Processing scripts to be executed by the SDP in response to requests.

• Helm Deployer Charts

  Charts used by the Helm deployer to deploy processing scripts and processing deployments.

• SDP notebooks

  Contains Jupyter notebooks that describe how one can control and monitor SDP via the console or using PyTango. Also contains notebooks to demonstrate SDP features. This module does not appear in the diagram, since it is a separate entity, not part of the SDP design. It uses BinderHub to connect to and interact with SDP deployed on a remote cluster.

• SDP Data Product Metadata

  It is a Python package to record SKA-specific metadata alongside data created by the SDP. It creates metadata files containing information defined by ADR-55. It is used by SDP workflows and pipelines that generate data products.

• SDP Data Queue Library

  Supports loading data on and off Kafka queues. In addition, it provides schemas for validating the data before sending to a queue. It is used by SDP components and pipelines to transfer and receive data via Kafka.

• SDP GSM

  Provides the ability to request Local Sky Models, and can host multiple versions of Global Sky Models.