Kubernetes Tutorials
This page aims to provide new developers with an overview of how K8S is used in MCCS development and deployment. It will cover the basics of deploying an MCCS cluster locally using Minikube, as well as how to deploy it to a remote Kubernetes cluster.
Deploy MCCS to a local Minikube cluster
Start Minikube
Minikube is a Kubernetes service, often used to teach basic K8S concepts and to run small-scale clusters. It’s the main choice for local development of MCCS software.
In order to install and set up Minikube, we can use the SKAO-provided project: ska-cicd-deploy-minikube.
git clone https://gitlab.com/ska-telescope/sdi/ska-cicd-deploy-minikube
Run the make command as follows:
cd ska-cicd-deploy-minikube
make DRIVER=docker MEM=16384 CPUS=8 all
Normally, the command is simply make all, but since the default container engine is Podman, we have to switch to docker (which is used by the MCCS pods).
Deploying an MCCS cluster to Minikube
For this tutorial, we will be deploying ska-low-mccs with minikube.
After setting up Minikube as explained in the previous step, it’s time to download the relevant repo and set it up:
git clone https://gitlab.com/ska-telescope/mccs/ska-low-mccs
git submodule init && git submodule update
The next step is to make sure to set docker as a driver (run this command whenever you start a new terminal). Here is a more detailed explanation as to what it does.
eval `minikube docker-env`
Now, to deploy the latest version of ska-low-mccs:
make k8s-install-chart
And to remove it:
make k8s-uninstall-chart
As a developer, you will have to deploy your local version of MCCS. To do so you will first need to create an OCI image:
make oci-build DRIVER=docker
This should give the following lines as output:
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 913B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 80B 0.0s
=> [internal] load metadata for artefact.skao.int/ska-build-python:0.1.1 0.1s
=> [internal] load metadata for artefact.skao.int/ska-tango-images-tango 0.1s
=> [internal] load build context 0.1s
=> => transferring context: 1.28MB 0.0s
=> [stage-1 1/10] FROM artefact.skao.int/ska-build-python:0.1.1@sha256: 0.0s
=> [tools 1/1] FROM artefact.skao.int/ska-tango-images-tango-dsconfig:1. 0.0s
=> CACHED [stage-1 2/10] COPY --from=tools /usr/local/bin/retry /usr/lo 0.0s
=> CACHED [stage-1 3/10] COPY --from=tools /usr/local/bin/wait-for-it.s 0.0s
=> CACHED [stage-1 4/10] WORKDIR /src 0.0s
=> [stage-1 5/10] COPY README.md pyproject.toml poetry.lock* ./ 0.1s
=> [stage-1 6/10] RUN poetry install --no-root 39.8s
=> [stage-1 7/10] COPY src ./ 0.1s
=> [stage-1 8/10] RUN poetry install 0.9s
=> [stage-1 9/10] COPY ./cache_data.py . 0.0s
=> [stage-1 10/10] RUN python ./cache_data.py 153.6s
=> exporting to image 9.1s
=> => exporting layers 9.1s
=> => writing image sha256:ad569fe021cf2b502cf06cf5900de215b28b4e11a0af3 0.0s
=> => naming to docker.io/library/ska-low-mccs:2.0.0-dirty 0.0s
Note the last line: naming to docker.io/library/ska-low-mccs:2.0.0-dirty. This line changes depending on software version and machine. The docker.io/library/ska-low-mccs:2.0.0-dirty string is made up of three parts: the registry, in this case, docker.io/library, the name: ska-low-mccs, and the tag 2.0.0-dirty
Navigate towards charts/ska-low-mccs/values.yaml and edit the following lines:
image:
registry: artefact.skao.int
name: ska-low-mccs
tag: ~
pullPolicy: IfNotPresent
Change the registry, name and tag to the values from the previous step. Save the file and then run make k8s-install-chart. You can also run make k8s-bounce to update a running deployment.
Run functional tests
As you might have noticed from the previous tutorial, building images and deploying them to a minikube cluster is fairly time-consuming. It’s recommended that you avoid it unless it’s necessary. In general, the workflow is simple: edit code, build the image and deploy as explained. Then run the testing pod:
make k8s-test
The test results are printed in the shell used to deploy the pod. To save the values to a file you can run the following:
script -q -c "make k8s-test" > k8s_test_results.txt
Run an interactive session with JupyTango
JupyTango provides an easy-to-use platform to work with an MCCS deployment.
me@local:~$ minikube ip
192.168.49.2
me@local:~$
Navigate to http://192.168.49.2/jupyterhub/user/mccs/lab/workspaces/auto-s for local development. (or wherever your persistent jupyterhub deployment is for non-local development)
import tango
db = tango.Database()
station_device_strings = db.get_device_exported("low-mccs/station/*")
stations = []
for device_str in station_device_strings:
device = tango.DeviceProxy(device_str)
device.adminMode = 0
stations.append(device)
# Do stuff with your stations here
MCCS has a selection of prebuilt notebooks (/notebooks), which can be loaded into your JupyTango session by clicking the ‘Upload’ button, if you then want to export your work, click ‘Download’.
Taranta
Taranta provides a Web UI for monitoring and controlling devices in the cluster. The MCCS charts have Taranta enabled by default, so once MCCS is deployed, you should be able to see Taranta in your web browser, at the cluster’s IP address.
Find out the IP address of the cluster:
Open a Web browser (Taranta works best with Chrome) and navigate to
http://192.168.49.2/ska-low-mccs/taranta/devices.
Log in with the credentials found here:
https://developer.skao.int/projects/ska-tango-taranta-suite/en/latest/taranta_users.html
Select the Dashboards tab on the left-hand side of the window.
#. You may now build your own dashboard, or import a dashboard from File. MCCS dashboards are available at ska-low-mccs/dashboards/.
Deploy MCCS to a remote Kubernetes cluster.
To deploy MCCS to a remote Kubernetes cluster, we generally use helmcharts and the process is a bit different. For this part of the tutorial, you need access to a remote cluster.
Configuring Kubectl
This part of the tutorial essentially sets up kubectl with the correct certificates and config values to connect to a remote cluster.
First, you will need to ssh into the server and locate the ~/.kube directory. This directory can usually be found in your user path (similar to ~/.ssh). If you can’t find it, contact the server admin and ask them to provide you with the kubeconfig file. (Or, if you are the admin, you can find them in /etc/kubernetes/admin.conf).
Use ssh copy to transfer them to your local machine:
scp user@remote-server:~/.kube/config ~/.kube/server-config
After this, you want to combine it with the local config file, so that you can use both local and remote clusters. Make sure to create a copy of your local kubeconfig file before running the following:
export KUBECONFIG=~/.kube/config:~/.kube/ral-config
kubectl ~/.kube/config view --flatten > ~/.kube/combined-config.yaml
Open the new file and double-check the values: cluster, user and context.
Deploying MCCS to a remote cluster
Create an SSH tunnel on the correct ports:
ssh -L 6448:<ip address>:6443 <user>@<ip address> -N
Make sure you have the correct context. You can check this by running:
kubectl config get-contexts
kubectl config use-context <your-context>
Finally, deploy an image:
helmfile -e < environment-name > sync
In general, all the helmfile environments are found in the helmfile.d/helmfile.yaml. They are customized for various servers used in developing and testing MCCS. Pick the relevant one.
Remove an image:
helmfile -e < environment-name > destroy
Deploying new images
In general, to test new code on servers you can utilize the pipeline images built for testing the code. Find the image tag in the oci-build step of the pipeline (a string of form 2.0.0-dev.cecae58fa)
Edit the relevant file in helmfile.d/values:
helm_values:
image:
registry: registry.gitlab.com/ska-telescope/mccs/ska-low-mccs
tag: <tag values>
Resource Usage
Resource usage for the MCCS pods is defined in ska-low-mccs/values.yaml. Currently, following a review of the resources required by the MCCS system, these are set to 20m CPU, and 50Mi memory, where 1000m is equivalent to 1 vCPU/Core for cloud providers, or 1 hyperthread on a bare-metal intel processor. Should these need to be increased in future, values.yaml will need updating. More information on resource usage in kubernetes can be found here: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
Charts
To view the charts avaliable in ska-low-mccs:
Navigate to https://artefact.skao.int/#browse/search=keyword%3Dska-low-mccs.
Open the helm release for the version of interest.
There are some optional tools avaliable for use:
- To deploy Jupyterhub add the folowing to the values file you are configuring the deployment with:
deploy-jupyterhub: true
- To deploy Taranta add the folowing to the values file you are configuring the deployment with:
deploy-taranta: true
An example helm values file is defined as follows: (found in /helmfile.d/mccs/values/stfc-ci.yaml)
overrides:
array:
stations:
"1":
id: 1
sps:
subracks:
"1":
simulated: true
srmb_host: srmb-1
srmb_port: 8081
tpms:
"10":
simulated: true
host: 10.0.10.201
port: 10000
version: tpm_v1_6
subrack: 1
subrack_slot: 1
pasd:
fndh:
gateway:
simulated: true
host: whatever
port: 9502
timeout: 10.0
controller:
modbus_id: 101
smartboxes:
"1":
fndh_port: 1
modbus_id: 1
"2":
fndh_port: 2
modbus_id: 2
antennas:
"100":
location_offset:
east: -3.25
north: 11.478
up: 0.023
eep: 100
smartbox: "1"
smartbox_port: 5
tpm: "1"
tpm_input: 5
"113":
location_offset:
east: -0.746
north: 12.648
up: 0.019
eep: 113
smartbox: "1"
smartbox_port: 7
tpm: "1"
tpm_input: 7
"2":
...
defaults:
logging_level_default: 5
subarrays:
"1":
enabled: true
logging_level_default: 4
subarraybeams:
"1": {}
"2": {}
"3": {}
"4": {}
The helm templates filter this environment-specific deployment yaml into the form: (found in /charts/ska-low-mccs/values.yaml)
deviceServers:
subarrays: # Tango device server type
subarray-01: # Tango device server instances
low-mccs/subarray/01: # Tango device instance TRL
subarray_id: 1
skuid_url: http://127.0.0.1:8000/
logging_level_default: 5
stationbeams: # Tango device server type
beam-001: # Tango device server instances
low-mccs/beam/001: # Tango device instance TRL
beam_id: 1
logging_level_default: 5