SKA-PST

This project provides the helm chart that includes the RECV.CORE, SMRB.CORE, and DSP.DISK software components of the Square Kilometre Array (SKA)’s Pulsar Timing (PST) instrument.

Architecture

Overview

The purpose of the PST (Pulsar Timing) is to process up to 16, dual-polarisation, oversampled, tied-array beams (from here on simply referred to as “beams”) formed in the SKA1 beamformers (CSP_Mid.CBF and CSP_Low.CBF), producing refined data products to be used in the SKA1 Science Data Processors (SDP_Mid and SDP_Low). The type of data product generated by the PST will depend on its configuration, with three specific observation modes supported: Pulsar Timing, Dynamic Spectrum and Flow Through. The observation mode is set on a per-beam basis and as such the PST is capable of producing multiple different types of data product simultaneously.

Artefacts

All of PST software components are built and packaged into artefacts as advised by the SKAO Developer Portal. The software binaries are packaged into OCI images while their deployment configurations are packaged into Helm artefacts.

OCI IMAGES

OCI Image Name

OCI Image Registry

Source Code Repository

ska-pst-dsp

artefact.skao.int/ska-pst-dsp

https://gitlab.com/ska-telescope/pst/ska-pst-dsp

ska-pst-lmc

artefact.skao.int/ska-pst-lmc

https://gitlab.com/ska-telescope/pst/ska-pst-lmc

ska-pst-recv

artefact.skao.int/ska-pst-recv

https://gitlab.com/ska-telescope/pst/ska-pst-recv

ska-pst-smrb

artefact.skao.int/ska-pst-smrb

https://gitlab.com/ska-telescope/pst/ska-pst-smrb

Helm Charts

The ska-pst is an umbrella chart consisting of ska-pst-core, and ska-pst-lmc sub-charts. See each of the components documentation pages for further information.

Software Components

PST Component Name

Short Description

Documentation link

DSP

Digital signal processing pipeline

https://developer.skao.int/projects/ska-pst-dsp/en/latest/

LMC

Local Monitoring and Control

https://developer.skao.int/projects/ska-pst-lmc/en/latest/

RECV

Data acquisition system

https://developer.skao.int/projects/ska-pst-recv/en/latest/

SMRB

Shared memory ring buffer manager

https://developer.skao.int/projects/ska-pst-smrb/en/latest/

Requirements

Kubernetes and Helm must be installed in order to run SKA PST.

Kubernetes

There are a number of ways to install and run Kubernetes as a local development environment. The one supported by the SKA is Minikube.

Helm

Helm is a kubernetes package manager. For its installation, see Installing Helm.

K9s

The default kubernetes client interface is kubectl. To simplify the interaction with kubernetes, K9s can be used. For more information, see k9scli.

Helm

This is a summary of the Helm chart parameters that can be used to customise the PST helm charts. The defaults values can be located in the chart’s values file.

SKA PST

Global values

Global values are used to propagate configuration to ska-pst-core and ska-pst-lmc.

Parameter

Description

Default

global.ports

PST.CORE components

dsp, recv, and smrb

global.ports.recv

RECV.CORE components.

recv-mgmt, and recv-datastream

global.ports.recv.recv-mgmt

Port configurations used for communication with PST.LMC

port, protocol, and targetport

global.ports.recv.recv-mgmt.port

Port exposed to outside of the k8s pod

18080

global.ports.recv.recv-mgmt.protocol

Protocol

TCP

global.ports.recv.recv-mgmt.targetport

Port exposed by a container in a k8s Pod

18080

global.ports.recv.recv-datastream

Port configurations used to recieve UDP data stream.

port, protocol, and targetport

global.ports.recv.recv-datastream.port

Port exposed to outside of the k8s pod

32080

global.ports.recv.recv-datastream.protocol

Protocol

UDP

global.ports.recv.recv-datastream.targetport

Port exposed by a container in a k8s Pod

32080

Override example:

# Override ports used for communication between ska-pst-core and ska-pst-lmc
global:
    tango_host: databaseds-tango-base-test:10000
    sub-system:
        ska-tango-base:
            enabled: false
    ports:
        recv:
            recv-mgmt:
                port:       28080
        smrb:
            smrb-mgmt:
                port:       28081
        dsp:
            dsp-mgmt:
                port:       28082

SKA PST CORE Configurations

Parameter

Description

Default

ska-pst-core.dsp.command

Command executed in the dsp container

bash

ska-pst-core.dsp.args

Arguments to the command

["-c","ska_pst_dsp_disk -v -c ${DSP_MGMT_PORT}"]

ska-pst-core.recv.command

Command executed in the recv container

bash

ska-pst-core.recv.args

Arguments to the command

["-c","ska_pst_smrb_core -v -c ${SMRB_MGMT_PORT}"]

ska-pst-core.smrb.command

Command executed in the smrb container

bash

ska-pst-core.smrb.args

Arguments to the command

["-c","ska_pst_recv_udpdb ${POD_IP} -v -p ${RECV_DATASTREAM_PORT} -c ${RECV_MGMT_PORT}"]

ska-pst-core.volumeMounts

List of storage configurations mounted to the dsp container of ska-pst-core

[{ containerPath: "/tmp/", name: "default", size: "500Mi", type: "emptyDir" }]

ska-pst-core.resources

Used for reserving k8s nodes resources. WARNING: supporting k8s infrastructure processes could be evicted as a result of resource starvation.

{}

Override example:

ska-pst-core:
    enabled: true
    # Override commands + args to bash to all containers in ska-pst-core
    recv:
        command: ["bash"]
        args: []
    smrb:
        command: ["bash"]
        args: []
    dsp:
        command: ["bash"]
        args: []

    # Use all available storage types in SKAO infrastructure
    # https://developer.skao.int/en/latest/tools/containers/orchestration-guidelines.html#storage-in-kubernetes-clusters-managed-by-the-systems-team
    volumeMounts:
        - containerPath: /tmp/
          name: default
          size: 50Gi
          type: emptyDir
        - containerPath: /mnt/hostPath
          hostPath: /tmp/
          name: mnt-hostPath
          type: hostPath
        - containerPath: /mnt/block
          name: mnt-block
          type: block # renders to k8s templates required for utilising storage class bds1
          size: 50Gi
        - containerPath: /mnt/nfs
          name: mnt-nfs
          type: nfs # renders to k8s templates required for utilising storage class nfss1
          size: 50Gi

    # Deploy to specific host. i.e. psi-pst1 in PSI LOW
    nodeSelector:
        kubernetes.io/hostname: psi-pst1

    # Request for specific k8s node resources
    limits:
      cpu: 1000m
      memory: 16Gi
    requests:
      cpu: 500m
      memory: 8Gi

SKA PST LMC Configurations

Detailed PST.LMC documentation is located at SKA PST LMC. Default ska-pst-lmc chart values are located at GitLab

Override example:

# Disable simulation mode
# Override PST.CORE component api endpoints to ska-pst-core
ska-pst-lmc:
    enabled: true
    beam:
        simulationMode: 0
    dsp:
        simulationMode: 0
        dsp_process_api_endpoint: ska-pst-core
    recv:
        simulationMode: 0
        recv_process_api_endpoint: ska-pst-core
    smrb:
        simulationMode: 0
        smrb_process_api_endpoint: ska-pst-core

SKA PST

Components

The ska-pst is an umbrella chart consisting of ska-pst-core, and ska-pst-lmc sub-charts.

Installation

Prerequisites

# Add SKA's helm repository
helm repo add skao https://artefact.skao.int/repository/helm-internal

# Search repository for ska-pst chart
helm search repo skao/ska-pst

# Store defaults to file
helm show values skao/ska-pst > values.yaml

# modify values (optional)
vi values.yaml

# Render template to provided values (optional)
## Used to validate if values are rendered to emplates as expected
make k8s-template-chart
## On the scenario that the SKA cicd makefile pipeline machinery isnt used.
helm template skao/ska-pst

Deployment

# Prepare environment variables (optional)
## Declare k8s namespace
export KUBE_NAMESPACE=pst
## Declare target application for wait
export KUBE_APP=ska-pst-lmc

# Install chart
## It can be installed using the SKA cicd makefile pipeline machinery.
make k8s-install-chart
## Environment variables can be used if overrides are required.
make K8S_CHART_PARAMS="--values=value.yaml" k8s-install-chart
## On the scenario that the SKA cicd makefile pipeline machinery isnt used.
helm install --generate-name skao/ska-pst
## And if overrides are desired
helm install --values=value.yaml --generate-name skao/ska-pst

# Wait for pods to be in a ready or completed state (optional)
## Use SKA cicd makefile pipeline machinery
make k8s-wait
## On the scenario that the SKA cicd makefile pipeline machinery isnt used.
kubectl -n ${KUBE_NAMESPACE} wait pod --for=condition=ready -l app=${KUBE_APP} --timeout=360s

SKA PST CORE STANDALONE

Overview

The ska-pst-core chart is used for deploying PST.CORE. It is used as a dependency chart by the ska-pst chart.

Components

PST.CORE consists of RECV.CORE, SMRB.CORE, and DSP.DISK. See each of the components documentation pages for further information.

Installation

Prerequisites

# Add SKA's helm repository
helm repo add skao https://artefact.skao.int/repository/helm-internal

# Search repository for ska-pst-core chart
helm search repo skao/ska-pst-core

# Store defaults to file
helm show values skao/ska-pst-core > values.yaml

# modify values (optional)
vi values.yaml

# Render template to provided values (optional)
## Used to validate if values are rendered to emplates as expected
make k8s-template-chart
## On the scenario that the SKA cicd makefile pipeline machinery isnt used.
helm template skao/ska-pst-core

Deployment

# Prepare environment variables (optional)
## Declare k8s namespace
export KUBE_NAMESPACE=pst
## Declare helm chart to be installed
export K8S_CHART=ska-pst-core
## Declare target application for wait
export KUBE_APP=ska-pst-core

# Install chart
## It can be installed using the SKA cicd makefile pipeline machinery.
make k8s-install-chart
## Environment variables can be used if overrides are required.
make K8S_CHART_PARAMS="--values=value.yaml" k8s-install-chart
## On the scenario that the SKA cicd makefile pipeline machinery isnt used.
helm install --generate-name skao/ska-pst-core
## And if overrides are desired
helm install --values=value.yaml --generate-name skao/ska-pst-core

# Wait for pods to be in a ready or completed state (optional)
## Use SKA cicd makefile pipeline machinery
make k8s-wait
## On the scenario that the SKA cicd makefile pipeline machinery isnt used.
kubectl -n ${KUBE_NAMESPACE} wait pod --for=condition=ready -l app=${KUBE_APP} --timeout=360s

Troubleshooting

Override files

To confirm compatibility of the values.yaml used for overriding the defaults, the following command can be used to confirm if the templates are rendered as expected.

helm template skao/ska-pst --values=values.yaml

Container logs

To specifically display the logs of all containers, application labels can be used to simplify the k8s command.

# Display logs of all containers in all pods that has the label ska-pst-core
kubectl -n pst logs -l app=ska-pst-core

# Display logs of all containers in all pods that has the label ska-pst-lmc
kubectl -n pst logs -l app=ska-pst-lmc

Automated Tests

Deployment

There are 3 charts present in the ska-pst repository. The test-parent, ska-pst and ska-pst-core. Each of the chart undergoes a series of deployment tests using GitLab’s CICD pipelines

Scan

Data Generator

The main tool for generating data is ska_pst_recv_udpgen which is an output artefact of the ska-pst-recv repository It is imported in the ska-pst-testutils oci image which resides in the ska-pst repository.

Simulating a Scan

The scan simulation is executed through the SKA cicd makefile pipeline machinery target k8s-test. The underlying code involves the usage of pytest and tango to interact with ska-pst-lmc and ska-pst-core. The simulation can be executed through the command make k8s-test PYTHON_VARS_TEST_SUITE=$TESTSUITE. The default test suite is integration.

Test Suite(s)

At this time of writing, there are currently two test suites. integration and invalidpackets tests. The tests are yet to be integrated with Jira and XRay where the test payload are in the form of Gherkin feature files to be ingested by pytest.

Sample Scan configuration JSON

Detailed description of the JSON schema has yet to be documented. The base configuration as well as overrides used for the scan simulation test can be found here.

Integration Tests

  • Single Scan

  • Multiple Scans

  • Abort

  • Abort before receiving packets

Negative Tests

  • Invalid Packets

Performance Tests

Performance related tests involving data transmission relies on the options available to ska_pst_recv_udpgen. The available options include the transmission rate (in Gb/s) and data transmission duration (in seconds).