================================== Set up your deployment environment ================================== The instructions at the :doc:`setup_development_environment` page will allow you to run tests in a lightweight test harness that mocks out Tango Controls. *This is sufficient to get you started developing.* However, running these tests does not actually deploy Tango devices. These instructions will allow you to set up a deployment environment so that you can run an actual cluster of Tango devices. They are largely independent of the instructions for setting up a development environment: you can follow these instructions to set up a deployment environment on the same machine as your development environment, or on a completely different machine. These instructions assume a Linux environment; it should be possible to deploy in other environments. Machine requirements -------------------- Memory requirements ^^^^^^^^^^^^^^^^^^^ As described below, MCCS uses the SKA `ska-cicd-deploy-minikube` project to manage cluster deployment. By default, `ska-cicd-deploy-minikube` requests 8Gb of memory for minikube. This implies that, assuming you want to be able to do other things with your computer while minikube is running, you will need upwards of 12Gb of memory. Actually, the MCCS chart currently requires only about 3Gb of memory, so if you need to deploy on a memory-constrained machine, it should be okay to overrule the `ska-cicd-deploy-minikube` default with a setting of 4Gb or even slightly less. Team members have managed to deploy on hardware with even less memory, but they were restricted to deploying minimal charts, and experienced difficulties running other applications at the same time. (Note that, operationally, the number of devices deployed by MCCS will increase over time by several orders of magnitude, eventually outstripping the capacity of any development machine. MCCS will maintain smaller "development" charts, that deploy a small number of representative devices, for development, demonstration and testing purposes. These charts are not expected to grow very much larger than they are at present, so it is probably safe to assume that an MCCS development chart will not require more than 4Gb of memory.) CPU requirements ^^^^^^^^^^^^^^^^ CPU requirements are much more forgiving; if you have less CPU available, MCCS will mostly just run more slowly. However, Tango commands timeout after three seconds (by default), and some MCCS commands currently take not much less than that to run. So if you deploy MCCS on a machine with slow or busy CPU, you may find that these commands exceed the three second limit, resulting in timeout issues. (These slow commands should not exist. MCCS plans improvements in this area. So in future, these timeouts will not be an issue.) By default, `ska-cicd-deploy-minikube` tells minikube to use two CPUs. As a rough rule of thumb: you probably won't see timeouts if minikube can get the two CPUs that it asks for; but if there is contention for those CPUs, you may see timeouts. Overview of setup / teardown ---------------------------- Installation and configuration of the cluster is handled by the SKA ``ska-cicd-deploy-minikube`` project. Thus, this project need only handle the deployment of our project charts to the cluster. The following state chart summarises the steps to deploying MCCS. Detailed instructions follow. .. uml:: setup_overview.uml Prerequisites and initial setup ------------------------------- As with all SKA subsystems, SKA Low MCCS uses Helm to deploy Kubernetes clusters of Docker containers. #. If you are setting up on a machine that has not already been set up with a development environment, then you will need to install Docker, install Git, and clone our repo; follow the instructions at :doc:`setup_development_environment`. #. Install kubectl. There are various ways to do this. On Ubuntu, one way is: .. code-block:: bash curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl chmod +x kubectl sudo mv ./kubectl /usr/local/bin/kubectl Don't be alarmed if running `kubectl` results in a config file error. A config file will be build the first time you run minikube. #. Clone the SKA ``ska-cicd-deploy-minikube`` project. (But first, if you have run minikube before, remove any old copies and their configuration files.) .. code-block:: bash find / -name minikube 2>/dev/null sudo rm -f rm -rf ~/.minikube .. code-block:: bash git clone git@gitlab.com:ska-telescope/sdi/ska-cicd-deploy-minikube.git Start the cluster manager ------------------------- #. Check for a new version of ``ska-cicd-deploy-minikube``. Development is ongoing, and you want to be running the latest version: .. code-block:: bash cd ~/ska-cicd-deploy-minikube git pull (Obviously there is no need to do this if you have only just cloned the project.) #. Use ``ska-cicd-deploy-minikube`` to install and configure the cluster: .. code-block:: bash make DRIVER=docker all // As we are using docker driver If deploying to a memory-constrained machine, the memory provided to minikube can be reduced from the 8Gb default: .. code-block:: bash make MEM=4096mb all The number of CPUs that minikube is allowed to use can also be changed from the default of 2: .. code-block:: bash make CPUS=4 all Note that to change these resource values on a cluster that has already been deployed, it must first be deleted: .. code-block:: bash make minikube-clean minikube-delete make MEM=16384mb CPUS=8 all #. **IMPORTANT** Because we are using docker as our driver, the environment must be set in your terminal. This command must be run in each new terminal: .. code-block:: bash eval $(minikube docker-env) Deploy MCCS to a cluster ------------------------ The basic steps to deploying MCCS are: #. Change into the ska-low-mccs directory, and build the development image ready for deployment to the cluster: .. code-block:: bash cd ~/ska-low-mccs make oci-build The ``make oci-build`` command must be rerun whenever the code is edited. The first time this command is run it can take a very long time because it has to download gigabytes of data. It may time out: just rerun it. #. Deploy the built image to the cluster. The basic command is .. code-block:: bash make k8s-install-chart This too may take a very long time the first time it is run. If you want to deploy the chart for a different environment, you need to set the K8S_HELMFILE_ENV variable as below: .. code-block:: bash make K8S_HELMFILE_ENV= k8s-install-chart You can find a list of available environments in /helmfile.d/helmfile.yaml. For PSI which is on a shared cluster it is also recommended to set the `RELEASE_NAME`: .. code-block:: bash make RELEASE_NAME=mccs-psi K8S_HELMFILE_ENV=psi-low k8s-install-chart #. Monitor the cluster to make sure it comes up okay. * To monitor the cluster yourself: .. code-block:: bash k9s After the image has been deployed to the cluster, you should see the device containers be created, and then the devices initialise. At first some devices may error; this is normal, and they will be automatically restarted. After several minutes, the cluster should stabilise and you will see that all devices are `Running` (except for the configuration pod, which will be `Completed`). If the k9s screen opens successfully but is completely empty there is a chance that you're experiencing a namespace issue, try pressing 0, and you should see all the pods appear. .. code-block:: shell-session $ make k8s-install-chart; k9s .. code-block:: shell-session $ make k8s-install-chart $ k9s # watch the cluster yourself using the k9s GUI application and exit when it is ready To use the deployed tango database with external tools from the cluster like jive or itango, it may be necessary to forward the port of the database. This can be done from the k9s screen. Navigate over the database pod and press .. code-block:: shell-session shift-f And press ok on the dialog box that appears. This should forward ports and allow the tango service to be used from outside Using the MCCS Deployment ------------------------- Now that the cluster is running, what can you do with it? See the :doc:`use_mccs` page for some options. Teardown MCCS ------------- Once you have finished with the deployment, you can tear it down: .. code-block:: bash make k8s-uninstall-chart Note that this does not teardown the minikube deployment, it simply unloads the MCCS charts. Teardown everything ------------------- There is no harm in leaving minikube running all the time. But if you `must` tear everything down, then .. code-block:: bash cd ~/ska-cicd-deploy-minikube make minikube-clean minikube-delete Install TANGO ------------- To set up TANGO, pogo, jive and all the other tools you will need to develop and test the MCCS project you can either use the install_script.sh or install_ansible.yml files that will install those and their dependancies for you. NOTE: These scripts should only be used with debian/ubuntu flavoured linux machines as they have not been tested with others and may not work If you already have an sql server installed then provide the password for it, otherwise choose a password that you would like for the server that will be installed. To use the shell script call .. code-block:: shell-session ./install_script.sh Because some tasks in the ansible playbook require sudo privileges you must call the ansible playbook with the --ask-become-pass, like this .. code-block:: shell-session ansible-playbook install_ansible.yml -e "SQL_PASSWORD=" --ask-become-pass alternativly you can follow the manual instructions here https://tango-controls.readthedocs.io/en/latest/installation/index.html