Resource Requirements

This document contains the PST resource requirements for the Array Assemblies (AA) for both the Low and Mid telescopes. In each case the requirements pertain to a single server that processes a single PST Beam. In the per AA configurations below PST includes support for Mid Bands 3 and 4, but these are not intended to be developed by SKA Mid, and so they are omitted from the configurations.

Software

For information on software dependencies, please see Software Requirements.

Common

The following requirements are common to all Array Assemblies.

Bare Metal Requirements

These configuration options should be adjusted by the administrators of the servers on which PST will be deployed. It is expected that these parameters would be configured via ansible playbooks. The following ansible playbooks can be used as reference for bootstrapping a PST server with the intention of joining a k8s cluster.

Link Description
Low PSI Ansible host vars used for PST specific servers in PSI Low environment
PST example ansible playbook PST managed example ansible playbook

Network Configuration

PST requires access to 100Gb Network Interface Cards (NIC) to receive the input data streams from the Low/Mid CBF and to transmit data products to the SDP. Ideally, there would be independent NICs for the CBF and SDP interfaces, but one shared NIC would be sufficient for early AAs, when the resource requirements from PST are lower. The configuration requirements for the NICs are:

  1. Input data must be received through a Mellanox 100Gb NIC, as PST utilises hardware offload features via the ibverbs library.

  2. The NIC (assuming interface name of $IFACE) should be configured with:

    1. Adaptive RX Coalescence Parameters off ethtool -C $IFACE adaptive-rx off

    2. Pause parameters disabled for RX and TX streams ethtool -A $IFACE rx off tx off

    3. Increased number of RX ring entries ethtool -G $IFACE rx 8192

  3. The input data streams from CBF will have packet sizes larger than the default Maximum Transmission Unit (MTU). Therefore all network devices will require the MTU to be increased, with the expected value being 9000 bytes.

CUDA Capable NVIDIA GPU

For AA1 onwards, PST requires an NVIDIA CUDA capable GPU that supports at least CUDA kernel version of 12.2 and driver version of 535.183.06.

Memory Locking Configuration

PST utilises memory locking to ensure that CPU RAM allocated to receive Ring Buffers cannot be swapped out when resource contention for memory is high. Memory locking also permits the CUDA driver to utilise Direct Memory Access (pinned transfers) between the CPU and GPU memory.

To increase the amount of lockable memory, the following configuration should be set in /etc/security/limits.conf.

soft memlock unlimited
hard memlock unlimited

Container Runtimes

PST signal processing for AA1 and beyond requires the Nvidia Container Runtime.

apiVersion: node.k8s.io/v1
handler: nvidia
kind: RuntimeClass
metadata:
  name: nvidia

Fast Local File System

PST requires access to a local file system (FS) with sufficient:

  • performance: read/write performance of at least 2 GB/s.

  • capacity: several TB, depending on the AA.

This local file system:

  • must be mounted at /data on the server onto which PST will be deployed; and

  • must have the appropriate access permissions (for UID=1000 and GID=1000) to permit containerised deployments with to read and write.

Kernel Configurations

PST requires changes to some default kernel configuration parameters

  • Disabling IPv6: net.ipv6.conf.all.disable_ipv6=1, net.ipv6.conf.default.disable_ipv6=1

  • Increase the default socket buffer sizes to 256 MB: net.core.rmem_max=268435456, net.core.wmem_max=268435456

  • Reverse path filtering: net.ipv4.conf.default.rp_filter=2

Bootstrap using the PST example ansible playbook.

Prerequisites

The baremetal configuration requirements can be automated through the PST example ansible playbook. It’s layout are as follows: example_playbook_layout

The following modifications are needed to target remote baremetal servers

  1. Update inventories/psi-low.yaml or add new inventory. i.e. touch inventories/new-pst-servers.yaml.

  2. Update or add host vars files in host_vars representing server specific configurations.

  3. Update or add group vars files in group_vars representing configurations propagated across pst servers.

  4. Install ansible specific python package. i.e. pip3 install ansible ansible-lint.

Ansible Execution

The following command can be used to execute ansible in check mode to determine the configurations to be implemented

ansible-playbook -i ./ansible/inventories/psi-low.yaml ./ansible/bootstrap-baremetal.yaml -C --diff

Sample outout is as follows alt text

Remove the -C to execute the example playbook and implement configurations.

Kubernetes

Memory Locking within a Kubernetes deployment

System services, such as the containerd service, will inherit the default memory locking configuration for the system, since PST requires memory locking within containers, the following modifications should be made to a containerd file such as: /etc/systemd/system/containerd.service.d/memlock.conf, with the content

[Service]
LimitMEMLOCK=infinity

Network Attachment Definition

A network attachment definition configuration is required to enable utilization of capabilities provided by Mellanox cards. Known working configuration is as follows:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: sample_nad
spec:
  config: |
    {
        "cniVersion": "0.3.1",
        "type": "macvlan",
        "master": "enp65s0f0", # enp65s0f0: name of the mellanox nic procured in bare metal
        "ipam": {
            "type": "whereabouts",
            "range": "192.168.121.0-192.168.121.19/24" # range should not conflict with k8s pod ip allocation range
        }
    }

Local Storage Configuration

Usage of the fast local file system is configured by overriding the primaryVolumeMount parameter in the PST helm chart, as shown in the example below. If this parameter is not over-ridden, then the PST chart will request storage from Kubernetes, but this will be a network storage type that will not support the performance or capacity requirements for PST.

ska-pst:
  core:
    primaryVolumeMount:
      type: local-storage
      hostPath: /data

AA0.5

For AA0.5, the PST Voltage Recorder mode will be deployed only at the Low telescope.

Requirement Low Mid B1 Mid B2 Mid B5
Number of Beams 1 0 0 0
Input Bandwidth [MHz] 75 N/A N/A N/A
Input Data Rate [Gb/s] 6.4 N/A N/A N/A
Max Output Data Rate [Gb/s] 6.4 N/A N/A N/A
Minimum CPU Cores 8 N/A N/A N/A
Ideal CPU Cores 8 N/A N/A N/A
Minimum CPU RAM [GB] 10 N/A N/A N/A
Ideal CPU RAM [GB] 32 N/A N/A N/A
FS Capacity [TB] 4+ N/A N/A N/A
FS Performance [GB/s] 2+ N/A N/A N/A

AA1

Mid.CBF will provide 4 x 198 MHz inputs in distinct sub-bands, with the maximum PST output rate limited to 1 x 198 MHz in Voltage Recorder mode.

Requirement Low Mid B1 Mid B2 Mid B5
Number of Beams 4 1 1 1
Input Bandwidth [MHz] 75 4x198 4x198 4x198
Input Data Rate [Gb/s] 6.4 57.9 57.9 57.9
Max Output Data Rate [Gb/s] 6.4 14.5 14.5 14.5
Minimum CPU Cores 8 8 8 8
Ideal CPU Cores 8 8 8 8
Minimum CPU RAM [GB] 10 60 60 60
Ideal CPU RAM [GB] 32 128 128 128
FS Capacity [TB] 4+ 4+ 4+ 4+
FS Performance [GB/s] 2+ 2+ 2+ 2+

AA2

Mid.CBF will transition from Talon to COTS hardware, providing the precise bandwidths listed in this table, which is a small reduction on the AA1 values for Bands 1 and 2. PST will drop the maximum required output data rate to 12.8 Gb/s for Mid bands.

Requirement Low Mid B1 Mid B2 Mid B5
Number of Beams 4 6 6 6
Input Bandwidth [MHz] 150 700 810 800
Input Data Rate [Gb/s] 12.8 51.2 59.2 58.5
Output Data Rate [Gb/s] 12.8 12.8 12.8 12.8
Minimum CPU Cores 8 8 8 8
Ideal CPU Cores 8 8 8 8
Minimum CPU RAM [GB] 10 60 60 60
Ideal CPU RAM [GB] 32 128 128 128
FS Capacity [TB] 4+ 4+ 4+ 4+
FS Performance [GB/s] 2+ 2+ 2+ 2+

AA*

Low.CBF’s rollout plan specifies 8 tied-array beams for AA*, with the full 16 not becoming available until AA4 (not a planned AA). Mid.CBF increase to 16 tied-array beams with the maximum supported bandwidth for Band 5 rising to 2500 MHz.

Requirement Low Mid B1 Mid B2 Mid B5
Number of Beams 8 16 16 16
Input Bandwidth [MHz] 300 700 810 2500
Input Data Rate [Gb/s] 25.6 51.2 59.2 91.4
Output Data Rate [Gb/s] 12.8 12.8 12.8 12.8
Minimum CPU Cores 16 16 16 16
Ideal CPU Cores 16 16 16 16
Minimum CPU RAM [GB] 32 128 128 128
Ideal CPU RAM [GB] 128 256 256 256
FS Capacity [TB] 4+ 4+ 4+ 4+
FS Performance [GB/s] 2+ 2+ 2+ 2+