SKA Test Equipment: Mid Sky Simulator Controller
Welcome to the SKA Mid Sky Simulator Controller project.
Getting started
Clone the project (Makefiles incl):
git clone --recurse-submodules git@gitlab.com:ska-telescope/ska-te-mid-skysimctl.git
(VSCode users) reopen the project in a devcontainer so that you have tango etc pre-installed. Once that’s done, from a terminal inside this running container, you can do any of the following.
Design & Implementation
The skysimctl is a service which runs on the Raspberry Pi in the Mid ITF. Currently the production version is running on the Pi0, but this will soon be changed to the Pi4. The skysimctl is implemented as a tango device, registered in the test-equipment tango DB. This means it can be controlled by users when they are connected to the test-equipment tango DB.
The skysimctl is used to control the signal devices by switching GPIO pins on/off. The GPIO pins are connected to relays which switch the signals on/off. It provides control over the following:
Band: Only one of Band1, Band2 & Band3 can be switched ON at any time.
Correlated_Noise_Source
Uncorrelated_Noise_Sources
H_Channelby
V_Channel
Each of these signals is implemented as a Tango r/w attribute.
Additionally, the Pi4 also has a humidity/temperature sensor installed. Currently, only the temperature is implemented and it is a read-only attribute. Whenever, the temperature is updated, if it exceeds the alarm threshold, the Correlated_Noise_Source and Uncorrelated_Noise_Sources are switched off in order to protect the sky simulator. Once the temperature drops back down to an acceptable level (i.e. below the warning threshold), they are switched back on. This process is visible in the following diagram:
Linting
This is useful for keeping your code up to standard.
make python-format && make python-lint
Testing
Always test your code, preferably locally. This command is the one that tests the python code on your local machine, and may require the use of mocks:
make python-test
Manual Testing
You can do manual testing against a local deployment running in minikube. In order to create the environment, you will need to have a running minikube, see https://gitlab.com/ska-telescope/sdi/ska-cicd-deploy-minikube. Once that has been setup, run the following:
eval $(minikube docker-env)
make oci-build-all
make k8s-install-chart
This deploys the test-parent
umbrella chart which includes the following:
skysimctl-simulator
: a skysimctl Tango device which does everything the actual device does, except interface with Raspberry Pi hardware.slack-webserver
: a test webserver which can receives messages from the alarm handleralarm-handler
: The alarm handler which monitors the temperature state of theskysimctl-simulator
and sends alerts to theslack-webserver
.
Deployment
Deployment of the Sky Simulator Controller is managed with Ansible. There are two deployment jobs in the CI/CD pipelines:
deploy-skysimctl-pi4
: Deploys the skysimctl service to the Raspberry Pi4 (10.165.3.15).deploy-webserver-pi4
: Deploys the skysim webserver to the Raspberry Pi4 (10.165.3.15).
These can be run locally using the deploy_skysimctl
or deploy_webserver
Make targets and setting PI_VERSION
to the required value.
For example:
TANGO_HOST=10.164.10.3:10000 PI_VERSION=pi4 make deploy_webserver
Raspberry Pi
Description
A Raspberry Pi with relay board is used as the physical implementation of the sky simulator controller. It runs a TCP server program (written in Python) that listens for and reacts to SCPI messages. For testing, any telnet client can be used. A TCP client has been developed (in Python) for automated testing. Currently there is one Raspberry Pi device:
Raspberry Pi4 (10.165.3.15) which is currently in development in the lab. This will replace the Pi0 in the future.
How to tango
Connecting to the device server
You can connect to the skysimctl device server from inside the Mid ITF VPN. You will need to set TANGO_HOST to the tango DB of the test-equipment. You can do this with the following command:
export TANGO_HOST=$(kubectl get -n test-equipment svc tango-databaseds -o jsonpath={'.status.loadBalancer.ingress[0].ip'}):10000
You can then connect to the skysimctl and use its DeviceProxy as follows:
$ itango3
ITango 9.3.3 -- An interactive Tango client.
Running on top of Python 3.10.12, IPython 7.31.1 and PyTango 9.3.3
help -> ITango's help system.
object? -> Details about 'object'. ?object also works, ?? prints more.
IPython profile: tango
hint: Try typing: mydev = Device("<tab>
In [1]: d = DeviceProxy("mid-itf/skysimctl/4")
In [2]: d.get_attribute_list()
Out[2]: ['temperature', 'humidity', 'All_Relay_States', 'Correlated_Noise_Source', 'H_Channel', 'V_Channel', 'Uncorrelated_Noise_Sources', 'Band', 'State', 'Status']
In [3]: d.H_Channel
Out[3]: False
In [4]: d.temperature
Out[4]: 24.2
In [5]: d.Band
Out[5]: <Band.BAND2: 2>
In [6]: d.Band = 3
In [7]: d.Band
Out[7]: <Band.BAND3: 3>
In [8]: d.Band = 2
Using tango admin
Check if the TAC (Tango Access Control) is enabled
$ ./build/tango_admin --tac-enabled
$ echo $?
0
Create server
tango_admin --add-server SkySimCtlDevice/pi4 SkySimCtlDevice mid-itf/skysimctl/4
Check server
$ tango_admin --check-server SkySimCtlDevice/pi4
$ echo $?
0
List servers
$ tango_admin --server-list
DataBaseds ProgAttenuatorDevice SignalGeneratorDevice SkySimCtlDevice SpectrumAnalyserDeviceAnritsu TangoAccessControl TangoRestServer TangoTest
List instances
$ tango_admin --server-instance-list SkySimCtlDevice
pi4
Taranta
Taranta page for skysimctl can be accessed here.
Alarm Handler
The Alarm Handler is used to send alerts to the atlas-itf-alerts Slack channel when the temperature read by the skysimctl exceeds certain thresholds.
Implementation
The alarm handler connects to a skysimctl DeviceProxy and subscribes to change events on the skysimctl’s temperature attribute. It uses the following logic to decide when to send alerts to Slack: When an event is received:
if it is an alarm or warning event:
if the last event we received is not the same event type, we fire an slack alert
if it is the same event type:
if more than 5 minutes have elapsed since the last alert, fire an slack alert otherwise, do nothing
if it is a valid event type:
if our previous event was an alarm or warning: fire a slack alert saying that the warning/alarm state has been cleared.
Deployment Details
The alarm handler is deployed with the deploy-alarm-handler
job. This deploys it into the test-equipment
namespace in the k8s cluster in the Mid ITF. It connects to the test-equipment
tango DB in order to open a connection to the skysimctl DeviceProxy.