SKA-PST-RECV

This project provides the libraries and applications required for the RECV software component of the Square Kilometre Array (SKA)’s Pulsar Timing (PST) instrument.

Documentation

Documentation Status

The documentation for this project, including the package description, Architecture description and the API modules can be found at SKA developer portal: https://developer.skao.int/projects/ska-pst-recv/en/latest/

Installation and Setup Instructions

First clone the repo and submodules to your local file system

git clone --recursive git@gitlab.com:ska-telescope/pst/ska-pst-recv.git

then change to the newly cloned directory and create the build/ sub-directory

cd ska-pst-recv
mkdir build

To simulate the build that will be performed in the Gitlab CI, a C++ builder image has been extended from the ska-cicd-cpp-build-base C++ building image. This image includes the required OS package and other custom software dependencies.

To launch this builder image as a container, run the following command.

make local-dev-env

When this command is executed, it will perform any required docker pull. Note that the current working directory will be mounted into the container as /mnt/ska-pst-recv

Building

The SKA-PST-RECV library and applications can be built in 3 ways.

Debug Build

The debug build will use the Cmake build arguments -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_FLAGS="-coverage" -DCMAKE_EXE_LINKER_FLAGS="-coverage". For debug purposes, the #define DEBUG will be defined for the software to enable debug features.

make .common.artifact-builder
make local-cpp-build-debug

Release Build

The release build will use the Cmake build arguments -DCMAKE_BUILD_TYPE=Release which ensures #define DEBUG is not defined. This build should be used for all deployments.

make .common.artifact-builder
make local-cpp-build-release

Linting Build

This build generates the compile_commands.json file which is used in the linting and static analysis tools. It uses clang++ as the compiler and sets the following Cmake build arguments -DCMAKE_BUILD_TYPE=Debug -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DCMAKE_CXX_COMPILER=clang++'.

make .common.artifact-builder
make .common.lint
make local-cpp-build-export-compile-commands

Documentation Build

Documentation is genereated using doxygen, which is then converted into ReadTheDocs format by Sphinx and Breathe. If these dependencies do not yet exist in ska-pst-recv-builder, then install them into the container

apt install -y doxygen
pip3 install sphinx sphinx-rtd-theme recommonmark
pip3 install -r docs/requirements.txt

The Documentation can then be build by

cd docs
make html

Linting

Linting is performed with 3 tools, cppcheck, clang-tidy and iwyu. Each of the 3 linting stages can be invidiually run with the following make targets:

make local-cpp-lint-clang 
make local-cpp-lint-iwyu
make local-cpp-lint-cppcheck

To perform all linting and generate the linting reports use the make target:

make local-cpp-lint

Building Docker Image

To build a Docker image for testing the applications

make OCI_IMAGE_BUILD_CONTEXT=${PWD} oci-build-all

There are quite a few options in here that are required to access the underlying hardware devices:

  1. Lock large amounts of shared memory

     --ulimit memlock=268435456:-1

  2. Access the host’s networking environment

     --net=host

  3. Use the network devices

     --device=/dev/infiniband/uverbs0 --device=/dev/infiniband/rdma_cm --device=/dev/infiniband/issm0 --device=/dev/infiniband/issm1 --device=/dev/infiniband/ucm0 --device=/dev/infiniband/ucm1 --device=/dev/infiniband/umad0 --device=/dev/infiniband/umad1 --device=/dev/infiniband/uverbs1

  4. Use the GPUs

     --gpus all

To run the docker image on pst-beam1 the following command line is required.

docker run -it --volume ${HOME}:/hosthome --ulimit memlock=-1  --net=host --entrypoint bash --ipc="host" --rm --gpus all --device=/dev/infiniband/uverbs0 --device=/dev/infiniband/uverbs1 --device=/dev/infiniband/uverbs2 --device=/dev/infiniband/uverbs3 --device=/dev/infiniband/rdma_cm --device=/dev/infiniband/issm0 --device=/dev/infiniband/issm1 --device=/dev/infiniband/issm2 --device=/dev/infiniband/issm3 --device=/dev/infiniband/umad0  --device=/dev/infiniband/umad1 --device=/dev/infiniband/umad2 --device=/dev/infiniband/umad3 --device=/dev/infiniband/uverbs1 --device=/dev/infiniband/uverbs0 --device=/dev/infiniband/uverbs2 --device=/dev/infiniband/uverbs3 ska-pst-recv:0.1.0-dirty

Then to execute the applications in the container (press CTRL+C to stop the application):

./ska_pst_recv_udprecv ${DATA_HOST} -i /usr/local/share/pst-beam1.txt

This will start the application and it should start printing the data capture statistics to standard output, once per second:

root@pst-beam1:/usr/local/bin# ./ska_pst_recv_udprecv ${DATA_HOST} -i /usr/local/share/pst-beam1.txt 
[2022-04-12 05:17:34.674] [info] Received	Dropped	Total Dropped	Sleeps
[2022-04-12 05:17:34.674] [info] 0	0	(0)	(0)
[2022-04-12 05:17:35.674] [info] 0	0	(0)	(0)
[2022-04-12 05:17:36.675] [info] 0	0	(0)	(0)

If the application cannot access the local infiniband devices, it will report an error:

root@pst-beam1:/usr/local/bin# ./ska_pst_recv_udprecv -i /usr/local/share/pst-beam1.txt
[2022-04-12 05:24:00.171] [error] Exception caught: ska::pst::recv::RDMACommunicationManager::bind rdma_bind_addr failed

Helm Chart

Helpers

  1. Render kubernetes manifest templates using default values from values.yaml

# Optional: specify target configuration payload in PrivateRules.mak
➜  ska-pst-recv git:(at3-246-expose-port-for-pod-communication) ✗ cat PrivateRules.mak  
K8S_CHART_PARAMS="--values=/mnt/$(PROJECT)/tests/integration/k8s-test/k8srunner.yaml"

➜  ska-pst-recv git:(at3-246-expose-port-for-pod-communication) ✗ make local_helm_render
docker run -ti --rm -v /Users/jesmigel/development/projects/swinburne/2022/ska/pi15/sprint4/ska-pst-recv:/mnt/ska-pst-recv -w /mnt/ska-pst-recv/charts/ska-pst-recv alpine/helm:3.8.2 template "--values=/mnt/ska-pst-recv/tests/integration/k8s-test/k8srunner.yaml" .
---
# Source: ska-pst-recv/templates/configmap-subband.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: "ska-pst-recv-configs-0"
  labels:
...

  1. Display contents of values.yaml

➜  ska-pst-recv git:(at3-246-expose-port-for-pod-communication) ✗ cat ./charts/ska-pst-recv/values.yaml
---
# Default values for ska-pst-recv.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

replicaCount: 1

enabled: true

hostNetwork: false
hostIPC: false

kind: Pod
...

Manual k8s deployment

  1. Set variables in local PrivateRules.mak

# i.e use specific values.yaml for helm chart as configuration payload
K8S_CHART_PARAMS="--values=tests/integration/k8s-test/k8srunner.yaml"
KUBE_NAMESPACE=demo

  1. Generate, review and deploy rendered manifest.yaml

#!/bin/bash
# Generate manifest.yaml
make k8s-template-chart

# Review manifest.yaml
cat manifest.yaml

# Deploy manifest.yaml
kubectl apply -f manifest.yaml

# Check deployment state
kubectl get -f manifest.yaml

# Grab ska-pst-recv container logs
kubectl logs -l app=ska-pst-recv --all-containers=true

Local k8s gitlab ci simulation

  1. Set variables in local PrivateRules.mak

# i.e use specific values.yaml for helm chart as configuration payload
K8S_CHART_PARAMS="--values=tests/integration/k8s-test/k8srunner-performance.yaml"
KUBE_NAMESPACE=ska-pst-recv

  1. Install python dependencies

pip3 install pytest pytest_bdd

  1. Execute bdd simulation

make ci_simulation_performance

Known bugs

None

Troubleshooting

None

License

See the LICENSE file for details.