Kubernetes Tutorials ==================== This page aims to provide new developers with an overview of how K8S is used in MCCS development and deployment. It will cover the basics of deploying an MCCS cluster locally using Minikube, as well as how to deploy it to a remote Kubernetes cluster. Deploy MCCS to a local Minikube cluster --------------------------------------- Start Minikube ^^^^^^^^^^^^^^ `Minikube `_ is a Kubernetes service, often used to teach basic K8S concepts and to run small-scale clusters. It's the main choice for local development of MCCS software. In order to install and set up Minikube, we can use the SKAO-provided project: `ska-cicd-deploy-minikube `_. .. code-block:: bash git clone https://gitlab.com/ska-telescope/sdi/ska-cicd-deploy-minikube Run the make command as follows: .. code-block:: bash cd ska-cicd-deploy-minikube make DRIVER=docker MEM=16384 CPUS=8 all Normally, the command is simply `make all`, but since the default container engine is Podman, we have to switch to docker (which is used by the MCCS pods). Deploying an MCCS cluster to Minikube ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ For this tutorial, we will be deploying `ska-low-mccs `_ with minikube. After setting up Minikube as explained in the previous step, it's time to download the relevant repo and set it up: .. code-block:: bash git clone https://gitlab.com/ska-telescope/mccs/ska-low-mccs git submodule init && git submodule update The next step is to make sure to set docker as a driver (run this command whenever you start a new terminal). Here is a more `detailed explanation as to what it does `_. .. code-block:: bash eval `minikube docker-env` Now, to deploy the latest version of ska-low-mccs: .. code-block:: bash make k8s-install-chart And to remove it: .. code-block:: bash make k8s-uninstall-chart As a developer, you will have to deploy your local version of MCCS. To do so you will first need to create an OCI image: .. code-block:: bash make oci-build DRIVER=docker This should give the following lines as output: .. code-block:: bash => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 913B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 80B 0.0s => [internal] load metadata for artefact.skao.int/ska-build-python:0.1.1 0.1s => [internal] load metadata for artefact.skao.int/ska-tango-images-tango 0.1s => [internal] load build context 0.1s => => transferring context: 1.28MB 0.0s => [stage-1 1/10] FROM artefact.skao.int/ska-build-python:0.1.1@sha256: 0.0s => [tools 1/1] FROM artefact.skao.int/ska-tango-images-tango-dsconfig:1. 0.0s => CACHED [stage-1 2/10] COPY --from=tools /usr/local/bin/retry /usr/lo 0.0s => CACHED [stage-1 3/10] COPY --from=tools /usr/local/bin/wait-for-it.s 0.0s => CACHED [stage-1 4/10] WORKDIR /src 0.0s => [stage-1 5/10] COPY README.md pyproject.toml poetry.lock* ./ 0.1s => [stage-1 6/10] RUN poetry install --no-root 39.8s => [stage-1 7/10] COPY src ./ 0.1s => [stage-1 8/10] RUN poetry install 0.9s => [stage-1 9/10] COPY ./cache_data.py . 0.0s => [stage-1 10/10] RUN python ./cache_data.py 153.6s => exporting to image 9.1s => => exporting layers 9.1s => => writing image sha256:ad569fe021cf2b502cf06cf5900de215b28b4e11a0af3 0.0s => => naming to docker.io/library/ska-low-mccs:2.0.0-dirty 0.0s Note the last line: `naming to docker.io/library/ska-low-mccs:2.0.0-dirty`. This line changes depending on software version and machine. The `docker.io/library/ska-low-mccs:2.0.0-dirty` string is made up of three parts: the registry, in this case, `docker.io/library`, the name: `ska-low-mccs`, and the tag `2.0.0-dirty` Navigate towards `charts/ska-low-mccs/values.yaml` and edit the following lines: .. code-block:: yaml image: registry: artefact.skao.int name: ska-low-mccs tag: ~ pullPolicy: IfNotPresent Change the registry, name and tag to the values from the previous step. Save the file and then run `make k8s-install-chart`. You can also run `make k8s-bounce` to update a running deployment. Run functional tests ^^^^^^^^^^^^^^^^^^^^ As you might have noticed from the previous tutorial, building images and deploying them to a minikube cluster is fairly time-consuming. It's recommended that you avoid it unless it's necessary. In general, the workflow is simple: edit code, build the image and deploy as explained. Then run the testing pod: .. code-block:: bash make k8s-test The test results are printed in the shell used to deploy the pod. To save the values to a file you can run the following: .. code-block:: bash script -q -c "make k8s-test" > k8s_test_results.txt Run an interactive session with JupyTango ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ JupyTango provides an easy-to-use platform to work with an MCCS deployment. .. code-block:: shell-session me@local:~$ minikube ip 192.168.49.2 me@local:~$ Navigate to http://192.168.49.2/jupyterhub/user/mccs/lab/workspaces/auto-s for local development. (or wherever your persistent jupyterhub deployment is for non-local development) .. code-block:: Python import tango db = tango.Database() station_device_strings = db.get_device_exported("low-mccs/station/*") stations = [] for device_str in station_device_strings: device = tango.DeviceProxy(device_str) device.adminMode = 0 stations.append(device) # Do stuff with your stations here MCCS has a selection of prebuilt notebooks (/notebooks), which can be loaded into your JupyTango session by clicking the 'Upload' button, if you then want to export your work, click 'Download'. Taranta ^^^^^^^ Taranta provides a Web UI for monitoring and controlling devices in the cluster. The MCCS charts have Taranta enabled by default, so once MCCS is deployed, you should be able to see Taranta in your web browser, at the cluster's IP address. #. Find out the IP address of the cluster: .. code-block:: shell-session me@local:~$ minikube ip 192.168.49.2 me@local:~$ #. Open a Web browser (Taranta works best with Chrome) and navigate to http://192.168.49.2/ska-low-mccs/taranta/devices. #. Log in with the credentials found here: https://developer.skao.int/projects/ska-tango-taranta-suite/en/latest/taranta_users.html #. Select the **Dashboards** tab on the left-hand side of the window. #. You may now build your own dashboard, or import a dashboard from File. MCCS dashboards are available at ska-low-mccs/dashboards/. Deploy MCCS to a remote Kubernetes cluster. ------------------------------------------- To deploy MCCS to a remote Kubernetes cluster, we generally use helmcharts and the process is a bit different. For this part of the tutorial, you need access to a remote cluster. Configuring Kubectl ^^^^^^^^^^^^^^^^^^^ This part of the tutorial essentially sets up kubectl with the correct certificates and config values to connect to a remote cluster. First, you will need to ssh into the server and locate the `~/.kube` directory. This directory can usually be found in your user path (similar to `~/.ssh`). If you can't find it, contact the server admin and ask them to provide you with the kubeconfig file. (Or, if you are the admin, you can find them in `/etc/kubernetes/admin.conf`). Use ssh copy to transfer them to your local machine: .. code-block:: bash scp user@remote-server:~/.kube/config ~/.kube/server-config After this, you want to combine it with the local config file, so that you can use both local and remote clusters. Make sure to create a copy of your local kubeconfig file before running the following: .. code-block:: bash export KUBECONFIG=~/.kube/config:~/.kube/ral-config kubectl ~/.kube/config view --flatten > ~/.kube/combined-config.yaml Open the new file and double-check the values: cluster, user and context. Deploying MCCS to a remote cluster ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Create an SSH tunnel on the correct ports: .. code-block:: bash ssh -L 6448::6443 @ -N Make sure you have the correct context. You can check this by running: .. code-block:: bash kubectl config get-contexts kubectl config use-context Finally, deploy an image: .. code-block:: bash helmfile -e < environment-name > sync In general, all the helmfile environments are found in the `helmfile.d/helmfile.yaml`. They are customized for various servers used in developing and testing MCCS. Pick the relevant one. Remove an image: .. code-block:: bash helmfile -e < environment-name > destroy Deploying new images ^^^^^^^^^^^^^^^^^^^^ In general, to test new code on servers you can utilize the pipeline images built for testing the code. Find the image tag in the oci-build step of the pipeline (a string of form `2.0.0-dev.cecae58fa`) Edit the relevant file in `helmfile.d/values`: .. code-block:: yaml helm_values: image: registry: registry.gitlab.com/ska-telescope/mccs/ska-low-mccs tag: Resource Usage -------------- Resource usage for the MCCS pods is defined in ska-low-mccs/values.yaml. Currently, following a review of the resources required by the MCCS system, these are set to 20m CPU, and 50Mi memory, where 1000m is equivalent to 1 vCPU/Core for cloud providers, or 1 hyperthread on a bare-metal intel processor. Should these need to be increased in future, values.yaml will need updating. More information on resource usage in kubernetes can be found here: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ Charts ------ To view the charts avaliable in ska-low-mccs: #. Navigate to https://artefact.skao.int/#browse/search=keyword%3Dska-low-mccs. #. Open the helm release for the version of interest. There are some optional tools avaliable for use: To deploy Jupyterhub add the folowing to the values file you are configuring the deployment with: .. code-block:: yaml deploy-jupyterhub: true To deploy Taranta add the folowing to the values file you are configuring the deployment with: .. code-block:: yaml deploy-taranta: true An example helm values file is defined as follows: (found in /helmfile.d/mccs/values/stfc-ci.yaml) .. code-block:: yaml overrides: array: stations: "1": id: 1 sps: subracks: "1": simulated: true srmb_host: srmb-1 srmb_port: 8081 tpms: "10": simulated: true host: 10.0.10.201 port: 10000 version: tpm_v1_6 subrack: 1 subrack_slot: 1 pasd: fndh: gateway: simulated: true host: whatever port: 9502 timeout: 10.0 controller: modbus_id: 101 smartboxes: "1": fndh_port: 1 modbus_id: 1 "2": fndh_port: 2 modbus_id: 2 antennas: "100": location_offset: east: -3.25 north: 11.478 up: 0.023 eep: 100 smartbox: "1" smartbox_port: 5 tpm: "1" tpm_input: 5 "113": location_offset: east: -0.746 north: 12.648 up: 0.019 eep: 113 smartbox: "1" smartbox_port: 7 tpm: "1" tpm_input: 7 "2": ... defaults: logging_level_default: 5 subarrays: "1": enabled: true logging_level_default: 4 subarraybeams: "1": {} "2": {} "3": {} "4": {} The helm templates filter this environment-specific deployment yaml into the form: (found in /charts/ska-low-mccs/values.yaml) .. code-block:: yaml deviceServers: subarrays: # Tango device server type subarray-01: # Tango device server instances low-mccs/subarray/01: # Tango device instance TRL subarray_id: 1 skuid_url: http://127.0.0.1:8000/ logging_level_default: 5 stationbeams: # Tango device server type beam-001: # Tango device server instances low-mccs/beam/001: # Tango device instance TRL beam_id: 1 logging_level_default: 5