Set up your deployment environment

The instructions at the Set up your development environment page will allow you to run tests in a lightweight test harness that mocks out Tango Controls. This is sufficient to get you started developing.

However, running these tests does not actually deploy Tango devices. These instructions will allow you to set up a deployment environment so that you can run an actual cluster of Tango devices. They are largely independent of the instructions for setting up a development environment: you can follow these instructions to set up a deployment environment on the same machine as your development environment, or on a completely different machine.

These instructions assume a Linux environment; it should be possible to deploy in other environments.

Machine requirements

Memory requirements

As described below, MCCS uses the SKA ska-cicd-deploy-minikube project to manage cluster deployment. By default, ska-cicd-deploy-minikube requests 8Gb of memory for minikube. This implies that, assuming you want to be able to do other things with your computer while minikube is running, you will need upwards of 12Gb of memory.

Actually, the MCCS chart currently requires only about 3Gb of memory, so if you need to deploy on a memory-constrained machine, it should be okay to overrule the ska-cicd-deploy-minikube default with a setting of 4Gb or even slightly less.

Team members have managed to deploy on hardware with even less memory, but they were restricted to deploying minimal charts, and experienced difficulties running other applications at the same time.

(Note that, operationally, the number of devices deployed by MCCS will increase over time by several orders of magnitude, eventually outstripping the capacity of any development machine. MCCS will maintain smaller “development” charts, that deploy a small number of representative devices, for development, demonstration and testing purposes. These charts are not expected to grow very much larger than they are at present, so it is probably safe to assume that an MCCS development chart will not require more than 4Gb of memory.)

CPU requirements

CPU requirements are much more forgiving; if you have less CPU available, MCCS will mostly just run more slowly.

However, Tango commands timeout after three seconds (by default), and some MCCS commands currently take not much less than that to run. So if you deploy MCCS on a machine with slow or busy CPU, you may find that these commands exceed the three second limit, resulting in timeout issues. (These slow commands should not exist. MCCS plans improvements in this area. So in future, these timeouts will not be an issue.)

By default, ska-cicd-deploy-minikube tells minikube to use two CPUs. As a rough rule of thumb: you probably won’t see timeouts if minikube can get the two CPUs that it asks for; but if there is contention for those CPUs, you may see timeouts.

Overview of setup / teardown

Installation and configuration of the cluster is handled by the SKA ska-cicd-deploy-minikube project. Thus, this project need only handle the deployment of our project charts to the cluster.

The following state chart summarises the steps to deploying MCCS. Detailed instructions follow.

Prerequisites and initial setup

As with all SKA subsystems, SKA Low MCCS uses Helm to deploy Kubernetes clusters of Docker containers.

If you are setting up on a machine that has not already been set up with a development environment, then you will need to install Docker, install Git, and clone our repo; follow the instructions at Set up your development environment.

Install kubectl. There are various ways to do this. On Ubuntu, one way is:

curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
chmod +x kubectl
sudo mv ./kubectl /usr/local/bin/kubectl

Don’t be alarmed if running kubectl results in a config file error. A config file will be build the first time you run minikube.

Clone the SKA ska-cicd-deploy-minikube project. (But first, if you have run minikube before, remove any old copies and their configuration files.)

find / -name minikube 2>/dev/null
sudo rm -f <path/minikube>
rm -rf ~/.minikube

git clone git@gitlab.com:ska-telescope/sdi/ska-cicd-deploy-minikube.git

Start the cluster manager

Check for a new version of ska-cicd-deploy-minikube. Development is ongoing, and you want to be running the latest version:
```
cd ~/ska-cicd-deploy-minikube
git pull
```
(Obviously there is no need to do this if you have only just cloned the project.)
Use ska-cicd-deploy-minikube to install and configure the cluster:
```
make DRIVER=docker all // As we are using docker driver
```
If deploying to a memory-constrained machine, the memory provided to minikube can be reduced from the 8Gb default:
make MEM=4096mb all
The number of CPUs that minikube is allowed to use can also be changed from the default of 2:
```
make CPUS=4 all
```
Note that to change these resource values on a cluster that has already been deployed, it must first be deleted:
```
make minikube-clean
minikube-delete
make MEM=16384mb CPUS=8 all
```
IMPORTANT Because we are using docker as our driver, the environment must be set in your terminal. This command must be run in each new terminal:
```
eval $(minikube docker-env)
```

Deploy MCCS to a cluster

The basic steps to deploying MCCS are:

Change into the ska-low-mccs directory, and build the development image ready for deployment to the cluster:
```
cd ~/ska-low-mccs
make oci-build
```
The make oci-build command must be rerun whenever the code is edited. The first time this command is run it can take a very long time because it has to download gigabytes of data. It may time out: just rerun it.
Deploy the built image to the cluster. The basic command is
```
make k8s-install-chart
```
This too may take a very long time the first time it is run.

If you want to deploy the chart for a different environment, you need to set the K8S_HELMFILE_ENV variable as below:
```
make K8S_HELMFILE_ENV=<YOUR_ENVIRONMENT> k8s-install-chart
```
You can find a list of available environments in /helmfile.d/helmfile.yaml. For PSI which is on a shared cluster it is also recommended to set the RELEASE_NAME:
```
make RELEASE_NAME=mccs-psi K8S_HELMFILE_ENV=psi-low k8s-install-chart
```
Monitor the cluster to make sure it comes up okay.
- To monitor the cluster yourself:
```
k9s
```
  After the image has been deployed to the cluster, you should see the device containers be created, and then the devices initialise. At first some devices may error; this is normal, and they will be automatically restarted. After several minutes, the cluster should stabilise and you will see that all devices are Running (except for the configuration pod, which will be Completed). If the k9s screen opens successfully but is completely empty there is a chance that you’re experiencing a namespace issue, try pressing 0, and you should see all the pods appear.
  $ make k8s-install-chart; k9s
```
$ make k8s-install-chart
$ k9s # watch the cluster yourself using the k9s GUI application and exit when it is ready
```
  To use the deployed tango database with external tools from the cluster like jive or itango, it may be necessary to forward the port of the database. This can be done from the k9s screen. Navigate over the database pod and press
  shift-f
  And press ok on the dialog box that appears. This should forward ports and allow the tango service to be used from outside

Using the MCCS Deployment

Now that the cluster is running, what can you do with it? See the Using an MCCS cluster page for some options.

Teardown MCCS

Once you have finished with the deployment, you can tear it down:

make k8s-uninstall-chart

Note that this does not teardown the minikube deployment, it simply unloads the MCCS charts.

Teardown everything

There is no harm in leaving minikube running all the time. But if you must tear everything down, then

cd ~/ska-cicd-deploy-minikube
make minikube-clean
minikube-delete

Install TANGO

To set up TANGO, pogo, jive and all the other tools you will need to develop and test the MCCS project you can either use the install_script.sh or install_ansible.yml files that will install those and their dependancies for you.

NOTE: These scripts should only be used with debian/ubuntu flavoured linux machines as they have not been tested with others and may not work

If you already have an sql server installed then provide the password for it, otherwise choose a password that you would like for the server that will be installed. To use the shell script call

./install_script.sh <your sql password>

Because some tasks in the ansible playbook require sudo privileges you must call the ansible playbook with the –ask-become-pass, like this

ansible-playbook install_ansible.yml -e "SQL_PASSWORD=<your sql password>" --ask-become-pass

alternativly you can follow the manual instructions here https://tango-controls.readthedocs.io/en/latest/installation/index.html