SKA-PST

This project provides the helm chart that includes the RECV.CORE, SMRB.CORE, and DSP.DISK software components of the Square Kilometre Array (SKA)’s Pulsar Timing (PST) instrument.

Architecture

Overview

The purpose of the PST (Pulsar Timing) is to process up to 16, dual-polarisation, oversampled, tied-array beams (from here on simply referred to as “beams”) formed in the SKA1 beamformers (CSP_Mid.CBF and CSP_Low.CBF), producing refined data products to be used in the SKA1 Science Data Processors (SDP_Mid and SDP_Low). The type of data product generated by the PST will depend on its configuration, with three specific observation modes supported: Pulsar Timing, Dynamic Spectrum and Flow Through. The observation mode is set on a per-beam basis and as such the PST is capable of producing multiple different types of data product simultaneously.

Artefacts

All of PST software components are built and packaged into artefacts as advised by the SKAO Developer Portal. The software binaries are packaged into OCI images while their deployment configurations are packaged into Helm artefacts.

OCI IMAGES

OCI Image Name	OCI Image Registry	Source Code Repository
ska-pst-dsp	artefact.skao.int/ska-pst-dsp	https://gitlab.com/ska-telescope/pst/ska-pst-dsp
ska-pst-lmc	artefact.skao.int/ska-pst-lmc	https://gitlab.com/ska-telescope/pst/ska-pst-lmc
ska-pst-recv	artefact.skao.int/ska-pst-recv	https://gitlab.com/ska-telescope/pst/ska-pst-recv
ska-pst-smrb	artefact.skao.int/ska-pst-smrb	https://gitlab.com/ska-telescope/pst/ska-pst-smrb

Helm Charts

The ska-pst is an umbrella chart consisting of ska-pst-core, and ska-pst-lmc sub-charts. See each of the components documentation pages for further information.

ska-pst

ska-pst-lmc

Software Components

PST Component Name	Short Description	Documentation link
DSP	Digital signal processing pipeline	https://developer.skao.int/projects/ska-pst-dsp/en/latest/
LMC	Local Monitoring and Control	https://developer.skao.int/projects/ska-pst-lmc/en/latest/
RECV	Data acquisition system	https://developer.skao.int/projects/ska-pst-recv/en/latest/
SMRB	Shared memory ring buffer manager	https://developer.skao.int/projects/ska-pst-smrb/en/latest/

Requirements

Kubernetes and Helm must be installed in order to run SKA PST.

Kubernetes

There are a number of ways to install and run Kubernetes as a local development environment. The one supported by the SKA is Minikube.

Helm

Helm is a kubernetes package manager. For its installation, see Installing Helm.

K9s

The default kubernetes client interface is kubectl. To simplify the interaction with kubernetes, K9s can be used. For more information, see k9scli.

Helm

This is a summary of the Helm chart parameters that can be used to customise the PST helm charts. The defaults values can be located in the chart’s values file.

SKA PST

Global values

Global values are used to propagate configuration to ska-pst-core and ska-pst-lmc.

Parameter	Description	Default
`global.ports`	PST.CORE components	`dsp`, `recv`, and `smrb`
`global.ports.recv`	RECV.CORE components.	`recv-mgmt`, and `recv-datastream`
`global.ports.recv.recv-mgmt`	Port configurations used for communication with PST.LMC	`port`, `protocol`, and `targetport`
`global.ports.recv.recv-mgmt.port`	Port exposed to outside of the k8s pod	18080
`global.ports.recv.recv-mgmt.protocol`	Protocol	TCP
`global.ports.recv.recv-mgmt.targetport`	Port exposed by a container in a k8s Pod	18080
`global.ports.recv.recv-datastream`	Port configurations used to recieve UDP data stream.	`port`, `protocol`, and `targetport`
`global.ports.recv.recv-datastream.port`	Port exposed to outside of the k8s pod	32080
`global.ports.recv.recv-datastream.protocol`	Protocol	UDP
`global.ports.recv.recv-datastream.targetport`	Port exposed by a container in a k8s Pod	32080

Override example:

# Override ports used for communication between ska-pst-core and ska-pst-lmc
global:
    tango_host: databaseds-tango-base-test:10000
    sub-system:
        ska-tango-base:
            enabled: false
    ports:
        recv:
            recv-mgmt:
                port:       28080
        smrb:
            smrb-mgmt:
                port:       28081
        dsp:
            dsp-mgmt:
                port:       28082

SKA PST CORE Configurations

Parameter	Description	Default
`ska-pst-core.dsp.command`	Command executed in the dsp container	bash
`ska-pst-core.dsp.args`	Arguments to the command	["-c","ska_pst_dsp_disk -v -c ${DSP_MGMT_PORT}"]
`ska-pst-core.recv.command`	Command executed in the recv container	bash
`ska-pst-core.recv.args`	Arguments to the command	["-c","ska_pst_smrb_core -v -c ${SMRB_MGMT_PORT}"]
`ska-pst-core.smrb.command`	Command executed in the smrb container	bash
`ska-pst-core.smrb.args`	Arguments to the command	["-c","ska_pst_recv_udpdb ${POD_IP} -v -p ${RECV_DATASTREAM_PORT} -c ${RECV_MGMT_PORT}"]
`ska-pst-core.volumeMounts`	List of storage configurations mounted to the dsp container of ska-pst-core	[{ containerPath: "/tmp/", name: "default", size: "500Mi", type: "emptyDir" }]
`ska-pst-core.resources`	Used for reserving k8s nodes resources. `WARNING: supporting k8s infrastructure processes could be evicted as a result of resource starvation.`	`{}`

Override example:

ska-pst-core:
    enabled: true
    # Override commands + args to bash to all containers in ska-pst-core
    recv:
        command: ["bash"]
        args: []
    smrb:
        command: ["bash"]
        args: []
    dsp:
        command: ["bash"]
        args: []

    # Use all available storage types in SKAO infrastructure
    # https://developer.skao.int/en/latest/tools/containers/orchestration-guidelines.html#storage-in-kubernetes-clusters-managed-by-the-systems-team
    volumeMounts:
        - containerPath: /tmp/
          name: default
          size: 50Gi
          type: emptyDir
        - containerPath: /mnt/hostPath
          hostPath: /tmp/
          name: mnt-hostPath
          type: hostPath
        - containerPath: /mnt/block
          name: mnt-block
          type: block # renders to k8s templates required for utilising storage class bds1
          size: 50Gi
        - containerPath: /mnt/nfs
          name: mnt-nfs
          type: nfs # renders to k8s templates required for utilising storage class nfss1
          size: 50Gi

    # Deploy to specific host. i.e. psi-pst1 in PSI LOW
    nodeSelector:
        kubernetes.io/hostname: psi-pst1

    # Request for specific k8s node resources
    limits:
      cpu: 1000m
      memory: 16Gi
    requests:
      cpu: 500m
      memory: 8Gi

SKA PST LMC Configurations

Detailed PST.LMC documentation is located at SKA PST LMC. Default ska-pst-lmc chart values are located at GitLab

Override example:

# Disable simulation mode
# Override PST.CORE component api endpoints to ska-pst-core
ska-pst-lmc:
    enabled: true
    beam:
        simulationMode: 0
    dsp:
        simulationMode: 0
        dsp_process_api_endpoint: ska-pst-core
    recv:
        simulationMode: 0
        recv_process_api_endpoint: ska-pst-core
    smrb:
        simulationMode: 0
        smrb_process_api_endpoint: ska-pst-core

SKA PST

Components

The ska-pst is an umbrella chart consisting of ska-pst-core, and ska-pst-lmc sub-charts.

Installation

Prerequisites

# Add SKA's helm repository
helm repo add skao https://artefact.skao.int/repository/helm-internal

# Search repository for ska-pst chart
helm search repo skao/ska-pst

# Store defaults to file
helm show values skao/ska-pst > values.yaml

# modify values (optional)
vi values.yaml

# Render template to provided values (optional)
## Used to validate if values are rendered to emplates as expected
make k8s-template-chart
## On the scenario that the SKA cicd makefile pipeline machinery isnt used.
helm template skao/ska-pst

Deployment

# Prepare environment variables (optional)
## Declare k8s namespace
export KUBE_NAMESPACE=pst
## Declare target application for wait
export KUBE_APP=ska-pst-lmc

# Install chart
## It can be installed using the SKA cicd makefile pipeline machinery.
make k8s-install-chart
## Environment variables can be used if overrides are required.
make K8S_CHART_PARAMS="--values=value.yaml" k8s-install-chart
## On the scenario that the SKA cicd makefile pipeline machinery isnt used.
helm install --generate-name skao/ska-pst
## And if overrides are desired
helm install --values=value.yaml --generate-name skao/ska-pst

# Wait for pods to be in a ready or completed state (optional)
## Use SKA cicd makefile pipeline machinery
make k8s-wait
## On the scenario that the SKA cicd makefile pipeline machinery isnt used.
kubectl -n ${KUBE_NAMESPACE} wait pod --for=condition=ready -l app=${KUBE_APP} --timeout=360s

SKA PST CORE STANDALONE

Overview

The ska-pst-core chart is used for deploying PST.CORE. It is used as a dependency chart by the ska-pst chart.

Components

PST.CORE consists of RECV.CORE, SMRB.CORE, and DSP.DISK. See each of the components documentation pages for further information.

Installation

Prerequisites

# Add SKA's helm repository
helm repo add skao https://artefact.skao.int/repository/helm-internal

# Search repository for ska-pst-core chart
helm search repo skao/ska-pst-core

# Store defaults to file
helm show values skao/ska-pst-core > values.yaml

# modify values (optional)
vi values.yaml

# Render template to provided values (optional)
## Used to validate if values are rendered to emplates as expected
make k8s-template-chart
## On the scenario that the SKA cicd makefile pipeline machinery isnt used.
helm template skao/ska-pst-core

Deployment

# Prepare environment variables (optional)
## Declare k8s namespace
export KUBE_NAMESPACE=pst
## Declare helm chart to be installed
export K8S_CHART=ska-pst-core
## Declare target application for wait
export KUBE_APP=ska-pst-core

# Install chart
## It can be installed using the SKA cicd makefile pipeline machinery.
make k8s-install-chart
## Environment variables can be used if overrides are required.
make K8S_CHART_PARAMS="--values=value.yaml" k8s-install-chart
## On the scenario that the SKA cicd makefile pipeline machinery isnt used.
helm install --generate-name skao/ska-pst-core
## And if overrides are desired
helm install --values=value.yaml --generate-name skao/ska-pst-core

# Wait for pods to be in a ready or completed state (optional)
## Use SKA cicd makefile pipeline machinery
make k8s-wait
## On the scenario that the SKA cicd makefile pipeline machinery isnt used.
kubectl -n ${KUBE_NAMESPACE} wait pod --for=condition=ready -l app=${KUBE_APP} --timeout=360s

Troubleshooting

Override files

To confirm compatibility of the values.yaml used for overriding the defaults, the following command can be used to confirm if the templates are rendered as expected.

helm template skao/ska-pst --values=values.yaml

Container logs

To specifically display the logs of all containers, application labels can be used to simplify the k8s command.

# Display logs of all containers in all pods that has the label ska-pst-core
kubectl -n pst logs -l app=ska-pst-core

# Display logs of all containers in all pods that has the label ska-pst-lmc
kubectl -n pst logs -l app=ska-pst-lmc

Automated Tests

Deployment

There are 3 charts present in the ska-pst repository. The test-parent, ska-pst and ska-pst-core. Each of the chart undergoes a series of deployment tests using GitLab’s CICD pipelines

Scan

Data Generator

The main tool for generating data is ska_pst_recv_udpgen which is an output artefact of the ska-pst-recv repository It is imported in the ska-pst-testutils oci image which resides in the ska-pst repository.

Simulating a Scan

The scan simulation is executed through the SKA cicd makefile pipeline machinery target k8s-test. The underlying code involves the usage of pytest and tango to interact with ska-pst-lmc and ska-pst-core. The simulation can be executed through the command make k8s-test PYTHON_VARS_TEST_SUITE=$TESTSUITE. The default test suite is integration.

Test Suite(s)

At this time of writing, there are currently two test suites. integration and invalidpackets tests. The tests are yet to be integrated with Jira and XRay where the test payload are in the form of Gherkin feature files to be ingested by pytest.

Sample Scan configuration JSON

Detailed description of the JSON schema has yet to be documented. The base configuration as well as overrides used for the scan simulation test can be found here.

Integration Tests

Single Scan
Multiple Scans
Abort
Abort before receiving packets

Negative Tests

Invalid Packets

Performance Tests

Performance related tests involving data transmission relies on the options available to ska_pst_recv_udpgen. The available options include the transmission rate (in Gb/s) and data transmission duration (in seconds).