Deploy to SKAO environments#

Configure a repository to deploy an application to a local or shared SKAO Kubernetes cluster.

Deploy to a local cluster#

SKAO repositories drive their builds with SKA pipeline machinery, included as a .make Git submodule. The same make targets run on your machine and in the CI pipeline, so you can deploy a chart to a local cluster with the commands the pipeline itself uses. Deploy a Minikube cluster first, as described in Set up a local Kubernetes environment.

Clone the repository with its submodules so that .make is present:

git clone --recurse-submodules <repository-url>

A project names its chart and namespace in its own Makefile and then includes the machinery:

KUBE_NAMESPACE ?= my-namespace
HELM_CHART ?= my-umbrella-chart
K8S_UMBRELLA_CHART_PATH ?= charts/$(HELM_CHART)/

include .make/k8s.mk
include .make/helm.mk
include .make/oci.mk
include .make/base.mk

Confirm which cluster and namespace the targets resolve to before you deploy anything:

make k8s-vars

Build the application image into the cluster’s Docker daemon when the chart uses a locally built image. Run this in a dedicated shell, because it repoints your Docker client at Minikube:

eval $(minikube docker-env)
make oci-build

Create the namespace, install the umbrella chart, and wait for the workloads to become ready:

make k8s-namespace
make k8s-install-chart
make k8s-wait

Inspect the deployment and run the repository’s tests inside the cluster. k8s-test copies the tests directory into a Pod and runs it against the live deployment, which is what the pipeline does:

make k8s-get
make k8s-podlogs
make k8s-test

Remove the deployment when you finish:

make k8s-uninstall-chart
make k8s-delete-namespace

Run make help for the available targets and make long-help for their variables. ska-tango-examples is a working reference for this layout.

Deploy to shared Kubernetes clusters#

SKAO publishes ready-made deployment jobs in the templates repository. Include the template for the environment you need instead of writing a deployment job yourself. Each template sets the namespace, the GitLab environment, and the runner tag for that environment type, so the naming stays consistent across projects.

Include the deployment jobs#

Add the umbrella include to .gitlab-ci.yml to get every deployment stage:

include:
  - project: 'ska-telescope/templates-repository'
    file: 'gitlab-ci/includes/deploy.gitlab-ci.yml'

Include a single template when you need one stage:

include:
  - project: 'ska-telescope/templates-repository'
    file: 'gitlab-ci/includes/k8s-test.gitlab-ci.yml'

Test a chart in a pipeline#

The k8s-test template deploys the chart, waits for the workloads, runs the repository’s tests inside the cluster, and removes the namespace again. It runs the same make targets you use locally:

k8s-test:
  image: $SKA_K8S_TOOLS_BUILD_DEPLOY
  stage: test
  tags:
    - ${SKA_K8S_RUNNER}
  variables:
    KUBE_NAMESPACE: 'ci-$CI_PROJECT_NAME-$CI_COMMIT_SHORT_SHA'
    CI_KEEP_NAMESPACE: $CLUSTER_NAMESPACE_MANAGER_RUNNING
  before_script:
    - make k8s-install-chart
    - make k8s-wait
  script:
    - make k8s-test || true
  environment:
    name: test/$CI_COMMIT_REF_SLUG
    on_stop: stop-k8s-test
    auto_stop_in: 1 minute
  rules:
    - exists:
        - tests/**/*

The job runs only when the repository contains a tests directory. Set CI_KEEP_NAMESPACE to true to keep the namespace after the job for investigation; otherwise the job uninstalls the chart and deletes the namespace. The paired stop-k8s-test job removes an environment left running.

Environment naming conventions#

Each template names its namespace and GitLab environment from the project and the Git reference. Do not set KUBE_NAMESPACE yourself unless you need a variant of one of these.

Environment type

Template

Namespace

GitLab environment

Test

k8s-test.gitlab-ci.yml

ci-<project>-<short-sha>

test/<branch-slug>

Development

deploy.dev.gitlab-ci.yml

ci-dev-<project>

ci-dev-<project>

Shared development

deploy.shared.gitlab-ci.yml

dev-shared-<branch-slug>

shared/<namespace>

Integration

deploy.integration.gitlab-ci.yml

integration-<project>

integration-<project>

Staging

deploy.staging.gitlab-ci.yml

staging-<project>

staging-<project>

Test namespaces are per-commit and short-lived. Development environments stop automatically after four hours. Integration and staging environments are persistent and named after the project alone, so each project has one of each.

Select a runner#

Deployment jobs select a runner through a variable rather than a literal tag:

tags:
  - ${SKA_K8S_RUNNER}

The runner determines which cluster the job deploys to, and it injects environment-specific variables into the job. Use the supplied variables rather than hard-coding a tag, so that a change of runner does not require a change to your pipeline. SKA_K8S_RUNNER defaults to ska-k8s, SKA_DEFAULT_RUNNER to ska-default, and SKA_GPU_RUNNER to ska-gpu-a100.

Find the available runners, their tags, and their resources in STFC Cloud Kubernetes Clusters.

Deploy to the AWS production cluster#

Deploying to the AWS production cluster needs permissions the System Team grants per application. Request them, then configure a job that uses the runner they assign.

  1. Ask the System Team for access. Open a System Team Support Desk, or ask in the #team-system Slack channel when the request needs discussion first. Tell them:

    • the name of the application,

    • the repository it is built from,

    • the namespace you want, following the prod-<app> convention.

  2. Wait for the kubeconfig assignment. The System Team generates a kubeconfig with the required permissions, assigns it to a GitLab runner, and gives you the runner tag. They also create the namespace.

  3. Configure the deployment job. In .gitlab-ci.yml, define a job tagged with the runner they assigned. Set K8S_SKIP_NAMESPACE to true, because k8s-install-chart-car invokes k8s-namespace and the assigned runner cannot create namespaces:

aws-deployment:
    stage: production
    tags:
        - ska-aws-runner-<project>
    variables:
        K8S_SKIP_NAMESPACE: true
        HELM_RELEASE: "<app>"
        K8S_CHART: "<app>-chart"
    script:
        - make k8s-install-chart-car
        - make k8s-wait
    environment:
        name: production
        kubernetes:
        namespace: "prod-<app>"

Access a cluster with InfraHQ#

InfraHQ grants short-lived Kubernetes credentials for the clusters that do not publish a public endpoint, including the ITF and Array Assembly environments. It writes a context into your kubeconfig, so kubectl, k9s, and a local Headlamp all work against the cluster once you have selected a destination.

Request access#

Access is granted per cluster, so request it before you need it.

Open a System Team Support Desk and state which clusters and namespaces you need. For production environments, ask in the #team-bang Slack channel instead.

Install the CLI#

Install the infra CLI for your platform:

# macOS
brew install infrahq/tap/infra

# Windows
scoop bucket add infrahq https://github.com/infrahq/scoop.git
scoop install infra

On Linux, download the package for your distribution from the Infra releases page and install it:

sudo dpkg -i infra_*_amd64.deb    # Debian and Ubuntu
sudo rpm -i infra_*_x86_64.rpm    # Fedora and RHEL

Confirm the CLI is available:

infra version

Sign in to the server#

Point KUBECONFIG at a separate file first, so that InfraHQ does not modify your existing Kubernetes configuration:

export KUBECONFIG=${HOME}/.kube/infra_config

Sign in with your SKAO identity:

infra login infra.skao.int

Set the server once in your shell profile if you prefer not to repeat it:

export INFRA_SERVER=infra.skao.int

Warning

infra login also accepts an access key through --key or INFRA_ACCESS_KEY. Treat an access key as a credential: never commit it to a repository or paste it into a ticket.

Select a cluster#

List the destinations your grants allow:

infra list

Select one to add its context to your kubeconfig and make it current. Append a namespace to land directly in it:

infra use <destination>
infra use <destination>.<namespace>

Confirm the connection#

Check which identity the cluster sees:

kubectl auth whoami
kubectl get pods -n <namespace>

These are the first commands that contact the cluster itself, so they distinguish two different failures.

A timeout or connection error means the cluster is unreachable from your network. Connect to the VPN if the cluster requires one, then try again.

A Forbidden error, or an empty infra list, means your grants do not cover that cluster or namespace. Return to Request access rather than retrying.

Run Headlamp locally#

Run Headlamp against the cluster you selected to get the same web interface the central datacentres publish. You need Docker, Podman, or another OCI runtime, and at least view access to the cluster’s resources.

Warning

Where the cluster is reachable only over a VPN, connect to the VPN before you use the connection and stay connected while Headlamp runs. Signing in to InfraHQ and selecting a destination can both succeed without the VPN, because they only contact the InfraHQ server. kubectl and Headlamp talk to the cluster itself, so they fail as soon as it is unreachable.

Give the container the path to the infra binary, which it calls to obtain credentials:

export INFRA_PATH=$(command -v infra)

Start Headlamp, mounting your home directory and the infra binary, and running as your own user so that it can read the kubeconfig InfraHQ wrote:

docker run -p 4466:4466 --name headlamp \
  -v /etc/passwd:/etc/passwd:ro \
  -v /etc/group:/etc/group:ro \
  -v ${HOME}:${HOME} -w ${HOME} -e HOME=${HOME} --user=$(id -u) \
  -v ${INFRA_PATH}:${INFRA_PATH} \
  -e KUBECONFIG=$KUBECONFIG \
  artefact.skao.int/ska-ser-headlamp-local:0.0.4

Find the current image tag on the ska-ser-headlamp-plugins releases page.

Open http://localhost:4466/ once the container is healthy. Headlamp lists every cluster your grants cover; select the one you want to inspect. From there, follow Diagnose a Kubernetes workload with Headlamp to diagnose a workload.

Remove the container when you finish:

docker rm -f headlamp

Next steps#

Once an application is deployed, observe and debug it from the tools documented elsewhere in the portal. Select the datacentre in each one, using the identifiers in SKAO environments reference.

Was this page helpful?