SKA Test Equipment: Mid Sky Simulator Controller

Welcome to the SKA Mid Sky Simulator Controller project.

Getting started

Clone the project (Makefiles incl):

git clone --recurse-submodules git@gitlab.com:ska-telescope/ska-te-mid-skysimctl.git

(VSCode users) reopen the project in a devcontainer so that you have tango etc pre-installed. Once that’s done, from a terminal inside this running container, you can do any of the following.

Design & Implementation

The skysimctl is a service which runs on the Raspberry Pi in the Mid ITF. Currently the production version is running on the Pi0, but this will soon be changed to the Pi4. The skysimctl is implemented as a tango device, registered in the test-equipment tango DB. This means it can be controlled by users when they are connected to the test-equipment tango DB.

The skysimctl is used to control the signal devices by switching GPIO pins on/off. The GPIO pins are connected to relays which switch the signals on/off. It provides control over the following:

  • Band: Only one of Band1, Band2 & Band3 can be switched ON at any time.

  • Correlated_Noise_Source

  • Uncorrelated_Noise_Sources

  • H_Channelby

  • V_Channel

Each of these signals is implemented as a Tango r/w attribute.

Additionally, the Pi4 also has a humidity/temperature sensor installed. Currently, only the temperature is implemented and it is a read-only attribute. Whenever, the temperature is updated, if it exceeds the alarm threshold, the Correlated_Noise_Source and Uncorrelated_Noise_Sources are switched off in order to protect the sky simulator. Once the temperature drops back down to an acceptable level (i.e. below the warning threshold), they are switched back on. This process is visible in the following diagram: Hysteresis

Linting

This is useful for keeping your code up to standard.

make python-format && make python-lint

Testing

Always test your code, preferably locally. This command is the one that tests the python code on your local machine, and may require the use of mocks:

make python-test

Manual Testing

You can do manual testing against a local deployment running in minikube. In order to create the environment, you will need to have a running minikube, see https://gitlab.com/ska-telescope/sdi/ska-cicd-deploy-minikube. Once that has been setup, run the following:

eval $(minikube docker-env)
make oci-build-all
make k8s-install-chart

This deploys the test-parent umbrella chart which includes the following:

  1. skysimctl-simulator: a skysimctl Tango device which does everything the actual device does, except interface with Raspberry Pi hardware.

  2. slack-webserver: a test webserver which can receives messages from the alarm handler

  3. alarm-handler: The alarm handler which monitors the temperature state of the skysimctl-simulator and sends alerts to the slack-webserver.

Deployment

Deployment of the Sky Simulator Controller is managed with Ansible. There are two deployment jobs in the CI/CD pipelines:

  1. deploy-skysimctl-pi4: Deploys the skysimctl service to the Raspberry Pi4 (10.165.3.15).

  2. deploy-webserver-pi4: Deploys the skysim webserver to the Raspberry Pi4 (10.165.3.15).

These can be run locally using the deploy_skysimctl or deploy_webserver Make targets and setting PI_VERSION to the required value. For example:

TANGO_HOST=10.164.10.3:10000 PI_VERSION=pi4 make deploy_webserver

Raspberry Pi

Description

A Raspberry Pi with relay board is used as the physical implementation of the sky simulator controller. It runs a TCP server program (written in Python) that listens for and reacts to SCPI messages. For testing, any telnet client can be used. A TCP client has been developed (in Python) for automated testing. Currently there is one Raspberry Pi device:

  1. Raspberry Pi4 (10.165.3.15) which is currently in development in the lab. This will replace the Pi0 in the future.

How to tango

Connecting to the device server

You can connect to the skysimctl device server from inside the Mid ITF VPN. You will need to set TANGO_HOST to the tango DB of the test-equipment. You can do this with the following command:

export TANGO_HOST=$(kubectl get -n test-equipment svc tango-databaseds -o jsonpath={'.status.loadBalancer.ingress[0].ip'}):10000

You can then connect to the skysimctl and use its DeviceProxy as follows:

$ itango3
ITango 9.3.3 -- An interactive Tango client.

Running on top of Python 3.10.12, IPython 7.31.1 and PyTango 9.3.3

help      -> ITango's help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.
IPython profile: tango
hint: Try typing: mydev = Device("<tab>

In [1]: d = DeviceProxy("mid-itf/skysimctl/4")
In [2]: d.get_attribute_list()
Out[2]: ['temperature', 'humidity', 'All_Relay_States', 'Correlated_Noise_Source', 'H_Channel', 'V_Channel', 'Uncorrelated_Noise_Sources', 'Band', 'State', 'Status']
In [3]: d.H_Channel
Out[3]: False
In [4]: d.temperature
Out[4]: 24.2
In [5]: d.Band
Out[5]: <Band.BAND2: 2>
In [6]: d.Band = 3
In [7]: d.Band
Out[7]: <Band.BAND3: 3>

In [8]: d.Band =  2

Using tango admin

Check if the TAC (Tango Access Control) is enabled

$ ./build/tango_admin --tac-enabled
$ echo $?
0

Create server

tango_admin --add-server SkySimCtlDevice/pi4 SkySimCtlDevice mid-itf/skysimctl/4

Check server

$ tango_admin --check-server SkySimCtlDevice/pi4
$ echo $?
0

List servers

$ tango_admin --server-list
DataBaseds ProgAttenuatorDevice SignalGeneratorDevice SkySimCtlDevice SpectrumAnalyserDeviceAnritsu TangoAccessControl TangoRestServer TangoTest

List instances

$ tango_admin --server-instance-list SkySimCtlDevice
pi4

Taranta

Taranta page for skysimctl can be accessed here.

Alarm Handler

The Alarm Handler is used to send alerts to the atlas-itf-alerts Slack channel when the temperature read by the skysimctl exceeds certain thresholds.

Implementation

The alarm handler connects to a skysimctl DeviceProxy and subscribes to change events on the skysimctl’s temperature attribute. It uses the following logic to decide when to send alerts to Slack: When an event is received:

  • if it is an alarm or warning event:

    • if the last event we received is not the same event type, we fire an slack alert

    • if it is the same event type:

    • if more than 5 minutes have elapsed since the last alert, fire an slack alert otherwise, do nothing

  • if it is a valid event type:

    • if our previous event was an alarm or warning: fire a slack alert saying that the warning/alarm state has been cleared.

Deployment Details

The alarm handler is deployed with the deploy-alarm-handler job. This deploys it into the test-equipment namespace in the k8s cluster in the Mid ITF. It connects to the test-equipment tango DB in order to open a connection to the skysimctl DeviceProxy.