# SKA Test Equipment: Mid Sky Simulator Controller

Welcome to the SKA Mid Sky Simulator Controller project.

## Getting started

Clone the project (Makefiles incl):

```bash
git clone --recurse-submodules git@gitlab.com:ska-telescope/ska-te-mid-skysimctl.git
```

(VSCode users) reopen the project in a devcontainer so that you have tango etc pre-installed. Once that's done, from a terminal inside this running container, you can do any of the following.

## Design & Implementation

The skysimctl is a service which runs on the Raspberry Pi in the Mid ITF. Currently the production version is running on the Pi0, but this will soon be changed to the Pi4. The skysimctl is implemented as a tango device, registered in the test-equipment tango DB. This means it can be controlled by users when they are connected to the test-equipment tango DB.

The skysimctl is used to control the signal devices by switching GPIO pins on/off. The GPIO pins are connected to relays which switch the signals on/off.
It provides control over the following:

* Band: Only one of Band1, Band2 & Band3 can be switched ON at any time.
* Correlated_Noise_Source
* Uncorrelated_Noise_Sources
* H_Channelby
* V_Channel

Each of these signals is implemented as a Tango r/w attribute.

Additionally, the Pi4 also has a humidity/temperature sensor installed. Currently, only the temperature is implemented and it is a read-only attribute. Whenever, the temperature is updated, if it exceeds the alarm threshold, the Correlated_Noise_Source and Uncorrelated_Noise_Sources are switched off in order to protect the sky simulator. Once the temperature drops back down to an acceptable level (i.e. below the warning threshold), they are switched back on. This process is visible in the following diagram:
![Hysteresis](static/images/hysteresis.png)

## Linting

This is useful for keeping your code up to standard.

```bash
make python-format && make python-lint
```

## Testing

Always test your code, preferably locally. This command is the one that tests the python code on your local machine, and may require the use of mocks:

```bash
make python-test
```

### Manual Testing

You can do manual testing against a local deployment running in minikube. In order to create the environment, you will need to have a running minikube, see <https://gitlab.com/ska-telescope/sdi/ska-cicd-deploy-minikube>. Once that has been setup, run the following:

```
eval $(minikube docker-env)
make oci-build-all
make k8s-install-chart
```

This deploys the `test-parent` umbrella chart which includes the following:

1. `skysimctl-simulator`: a skysimctl Tango device which does everything the actual device does, except interface with Raspberry Pi hardware.
2. `slack-webserver`: a test webserver which can receives messages from the alarm handler
3. `alarm-handler`: The alarm handler which monitors the temperature state of the `skysimctl-simulator` and sends alerts to the `slack-webserver`.

## Deployment

Deployment of the Sky Simulator Controller is managed with Ansible. There are two deployment jobs in the CI/CD pipelines:

1. `deploy-skysimctl-pi4`: Deploys the skysimctl service to the Raspberry Pi4 (10.165.3.15).
2. `deploy-webserver-pi4`: Deploys the skysim webserver to the Raspberry Pi4 (10.165.3.15).

These can be run locally using the `deploy_skysimctl` or `deploy_webserver` Make targets and setting `PI_VERSION` to the required value.
For example:

```bash
TANGO_HOST=10.164.10.3:10000 PI_VERSION=pi4 make deploy_webserver
```

## Raspberry Pi

### Description

A Raspberry Pi with relay board is used as the physical implementation of the sky simulator controller. It runs a TCP server program (written in Python) that listens for and reacts to SCPI messages. For testing, any telnet client can be used. A TCP client has been developed (in Python) for automated testing. Currently there is one Raspberry Pi device:

1. Raspberry Pi4 (10.165.3.15) which is currently in development in the lab. This will replace the Pi0 in the future.

## How to tango

### Connecting to the device server

You can connect to the skysimctl device server from inside the Mid ITF VPN. You will need to set TANGO_HOST to the tango DB of the test-equipment.
You can do this with the following command:

```bash
export TANGO_HOST=$(kubectl get -n test-equipment svc tango-databaseds -o jsonpath={'.status.loadBalancer.ingress[0].ip'}):10000
```

You can then connect to the skysimctl and use its DeviceProxy as follows:

```
$ itango3
ITango 9.3.3 -- An interactive Tango client.

Running on top of Python 3.10.12, IPython 7.31.1 and PyTango 9.3.3

help      -> ITango's help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.
IPython profile: tango
hint: Try typing: mydev = Device("<tab>

In [1]: d = DeviceProxy("mid-itf/skysimctl/4")
In [2]: d.get_attribute_list()
Out[2]: ['temperature', 'humidity', 'All_Relay_States', 'Correlated_Noise_Source', 'H_Channel', 'V_Channel', 'Uncorrelated_Noise_Sources', 'Band', 'State', 'Status']
In [3]: d.H_Channel
Out[3]: False
In [4]: d.temperature
Out[4]: 24.2
In [5]: d.Band
Out[5]: <Band.BAND2: 2>
In [6]: d.Band = 3
In [7]: d.Band
Out[7]: <Band.BAND3: 3>

In [8]: d.Band =  2
```

### Using tango admin

Check if the TAC (Tango Access Control) is enabled

```bash
$ ./build/tango_admin --tac-enabled
$ echo $?
0
```

Create server

```bash
tango_admin --add-server SkySimCtlDevice/pi4 SkySimCtlDevice mid-itf/skysimctl/4
```

Check server

```bash
$ tango_admin --check-server SkySimCtlDevice/pi4
$ echo $?
0
```

List servers

```bash
$ tango_admin --server-list
DataBaseds ProgAttenuatorDevice SignalGeneratorDevice SkySimCtlDevice SpectrumAnalyserDeviceAnritsu TangoAccessControl TangoRestServer TangoTest
```

List instances

```bash
$ tango_admin --server-instance-list SkySimCtlDevice
pi4
```

## Taranta

Taranta page for skysimctl can be accessed [here](https://k8s.miditf.internal.skao.int/test-equipment/taranta/devices).

## Alarm Handler

The Alarm Handler is used to send alerts to the atlas-itf-alerts Slack channel when the temperature read by the skysimctl exceeds certain thresholds.

### Implementation

The alarm handler connects to a skysimctl DeviceProxy and subscribes to change events on the skysimctl's temperature attribute.
It uses the following logic to decide when to send alerts to Slack:
When an event is received:

* if it is an alarm or warning event:
  * if the last event we received is not the same event type, we fire an slack alert
  * if it is the same event type:
  * if more than 5 minutes have elapsed since the last alert, fire an slack alert otherwise, do nothing
* if it is a valid event type:
  * if our previous event was an alarm or warning: fire a slack alert saying that the warning/alarm state has been cleared.

### Deployment Details

The alarm handler is deployed with the `deploy-alarm-handler` job. This deploys it into the `test-equipment` namespace in the k8s cluster in the Mid ITF. It connects to the `test-equipment` tango DB in order to open a connection to the skysimctl DeviceProxy.