.. _design_modules:

Modules
=======

.. figure:: ../images/sdp_modules.svg
  :align: center

  SDP software modules.

The SDP is built from software modules which produce a number of different
types of artefacts. The components of the system are built as Open Container
Initiative (OCI) images which are deployed on a Kubernetes cluster using a Helm
chart. The OCI images depend on libraries containing common code. The diagram
shows the SDP modules and the dependencies between them.

The source code is hosted in the `SKA Science Data Processor group in GitLab
<https://gitlab.com/ska-telescope/sdp>`_, in the following repositories:

- `Integration <https://gitlab.com/ska-telescope/sdp/ska-sdp-integration>`_

  Integration of components into the SDP system. Contains the Helm chart to
  deploy the SDP, the system integration tests, and this documentation.

- `Configuration Library <https://gitlab.com/ska-telescope/sdp/ska-sdp-config>`_

  Library providing the interface to the configuration database.

- `Etcd <https://gitlab.com/ska-telescope/sdp/ska-sdp-etcd>`_

  Etcd is the highly available key-value database underpinning the configuration
  library.

- `Local Monitoring and Control <https://gitlab.com/ska-telescope/sdp/ska-sdp-lmc>`_

  Tango devices providing the local monitoring and control (LMC) of the SDP.

- `LMC Queue Connector <https://gitlab.com/ska-telescope/sdp/ska-sdp-lmc-queue-connector>`_

  The Queue Connector Tango devices provide the interface for exchanging real-time
  data with other telescope subsystems.

- `Processing Controller <https://gitlab.com/ska-telescope/sdp/ska-sdp-proccontrol>`_

  The Processing Controller controls the execution of processing blocks.

- `Buffer Manager <https://gitlab.com/ska-telescope/sdp/ska-sdp-buffermanager>`_

  The Buffer Manager manages the buffer space, clearing up old data as needed.

- `Helm Deployer <https://gitlab.com/ska-telescope/sdp/ska-sdp-helmdeploy>`_

  The Helm Deployer provides the interface between the system and the Kubernetes platform.

- `Slurm Deployer <https://gitlab.com/ska-telescope/sdp/ska-sdp-slurmdeploy>`_

  The Slurm Deployer sends slurm jobs to a Slurm cluster.

- `Console <https://gitlab.com/ska-telescope/sdp/ska-sdp-console>`_

  Command-line interface to control and monitor the SDP.

- `Processing Scripting Library <https://gitlab.com/ska-telescope/sdp/ska-sdp-scripting>`_

  Library providing the high-level interface for writing processing scripts.

- `Processing Scripts <https://gitlab.com/ska-telescope/sdp/ska-sdp-script>`_

  Processing scripts to be executed by the SDP in response to requests.

- `Helm Deployer Charts
  <https://gitlab.com/ska-telescope/sdp/ska-sdp-helmdeploy-charts>`_

  Charts used by the Helm deployer to deploy processing scripts and processing
  deployments.

- `SDP notebooks <https://gitlab.com/ska-telescope/sdp/ska-sdp-notebooks>`_

  Contains Jupyter notebooks that describe how one can control and monitor SDP
  via the console or using PyTango. Also contains notebooks to demonstrate SDP
  features. This module does not appear in the diagram, since it is a separate
  entity, not part of the SDP design. It uses BinderHub to connect to and
  interact with SDP deployed on a remote cluster.

- `SDP Data Product Metadata <https://gitlab.com/ska-telescope/sdp/ska-sdp-dataproduct-metadata>`_

  It is a Python package to record SKA-specific metadata alongside data created by the SDP.
  It creates metadata files containing information defined by
  `ADR-55 <https://confluence.skatelescope.org/display/SWSI/ADR-55+Definition+of+metadata+for+data+management+at+AA0.5>`_.
  It is used by SDP workflows and pipelines that generate data products.

- `SDP Data Queue Library <https://gitlab.com/ska-telescope/sdp/ska-sdp-dataqueues>`_

  Supports loading data on and off Kafka queues. In addition, it provides schemas for validating
  the data before sending to a queue. It is used by SDP components and pipelines to transfer
  and receive data via Kafka.

- `SDP GSM <https://gitlab.com/ska-telescope/sdp/ska-sdp-global-sky-model>`_

  Provides the ability to request Local Sky Models, and can host multiple
  versions of Global Sky Models.