SKAMPI Deployment Pipelines

This section focuses on deploying the SKAMPI suite of software products for testing and integration using the Gitlab CI Pipelines. If you have enough resources it is possible to deploy the MVP locally or on a dedicated server using a minikube installation compatible with SKAMPI like the one available from SKA Minikube deployment. Also, it is recommended that the SKA Tango Operator is deployed. Nonetheless Gitlab CI pipelines is the recommended method since the clusters are configured to support all of the required SKAMPI features.

When dealing with the CI SKAMPI infrastructure it is important to distinguish between:

  • Branch and Merge Request (MR) pipelines (development)

  • Main Branch pipelines (integration)

  • Tag pipelines (publishing and staging)

Pipelines will be using needs keyword to speed up development workflow and break-up unnecessary job dependencies between jobs. This means that stages here are for illustrative purposes and any job without any prior dependency, for example docs-build, will start as soon as the pipeline starts.

The development pipeline is split up in two phases: on-demand deployments and the automated deployment and test of SKAMPI. The testing pipelines and code is is ska-skampi, which means we will trigger downstream pipelines to run our tests.

Here is how the on-demand jobs look like:

_images/on_demand_pipeline.png

Note

Due to issues of long SKAMPI branch names breaking the deployments, we’ve moved the on-demand naming to be ci-ska-skampi-<pipeline id>-$CONFIG. Please be mindful to re-use the same pipeline whenever possible and to cleanup an on-demand environment that you no longer need. They will be cleaned up after 2 hours, if you forget.

As we can see, we have several stages, splitted by datacentre, were we have low and mid deployment, information, test and destroy jobs. Also, we have a redeploy job to recreate the environment from scratch.

These jobs allow us to manage and test a SKAMPI deployment:

  • deploy - Deploys the environment to a namespace named after the gitlab pipeline. This is important as in the same pipeline, each job (given datacentre and config) will target the same namespace

  • destroy - Undeploys SKAMPI and removes the namespaces, along with their resources

  • info - Collects information on the contents of the namespace (i.e., pods, images, etc)

  • redeploy - Undeploys and deploys a fresh environment, in the same namespace

  • test - Launches a test pipeline against the on-demand environment. The scope of the tests can be changed via environment variables of the job. To know more, check the testing documentation

Also, with each change, we must deploy both telescopes in a fresh environment and run tests against them, so that SKAMPI changes are always verified for compliance.

_images/deploy_test_pipeline.png

After linting and building, the pipeline will deploy test environments, trigger test pipelines against them, and clean up afterwards. Currently, the cleanup task only runs upon a succesful test, to let developers debug an environment were tests failed.

The test jobs deploys the charts in pipeline-defined unique namespaces and run the tests directly on the runner pods, eliminating overhead and enabling easy integration. In the Kubernetes Clusters maintained by the System Team branch namespaces are deleted after a timeout (in hours) to prevent failing branch pipelines to accumulate.

The test-low and test-mid jobs run verification tests covering all components on both telescopes. Developers must be aware that configuration of the tests should be provided only by Makefiles and that all jobs for MID and LOW should be aligned so that they look the same except for the configuration. Pipeline Variables, CI/CD Variables and command line options should not be used without defining the variables in Makefiles first, for visibility.

We should avoid setting Helm chart variables using the Makefile, and resort to using custom values files. The reason for this is because variables are less visible and can override settings set by values files. The exception are GitLab’s variables such as CI_COMMIT_SHA, but even for those there is a make target to set them as environment variables so that local environments are same as the pipelines.

Aside from the on-demand and testing parts of the pipeline, we also have the capability to deploy SKAMPI on long-lived environments.

_images/integration_staging.png

This depicts the dependencies we have for integration and staging pipelines. Again, splitted by datacentre (in reality PSIs and ITFs are different datacentres, but they were joined together to save space in the Gitlab UI), we have at the top of the dependencies, the testing on the telescope. We cannot deploy to integration if the tests have not passed for each telescope. Likewise, to deploy to the staging environment, the deployment into integration must have been successful.

The SKAMPI repository folder structure is as follows:

_images/folders.png

This follows standard SKAO practices: a charts folder and a documentation folder, also containing custom scripts and resources. The SKAO pipeline machinery can thus be used and the .gitlab-ci.yml file will include the following lines:

include:
  # Lint and build
  - project: "ska-telescope/templates-repository"
    file: "gitlab-ci/includes/helm-chart.gitlab-ci.yml"

  - project: "ska-telescope/templates-repository"
    file: "gitlab-ci/includes/docs-build.gitlab-ci.yml"

  # Deployments
  - local: "/.gitlab/ci/common.gitlab-ci.yml"
  - local: "/.gitlab/ci/deploy-test.gitlab-ci.yml"
  - local: "/.gitlab/ci/low-itf.gitlab-ci.yml"
  - local: "/.gitlab/ci/mid-itf.gitlab-ci.yml"
  - local: "/.gitlab/ci/psi-low.gitlab-ci.yml"
  - local: "/.gitlab/ci/psi-mid.gitlab-ci.yml"
  - local: "/.gitlab/ci/stfc-techops.gitlab-ci.yml"
  - local: "/.gitlab/ci/stfc-dp.gitlab-ci.yml"

  # .post steps
  - project: "ska-telescope/templates-repository"
    file: "gitlab-ci/includes/finaliser.gitlab-ci.yml"

Nonetheless some customizations are required. The .gitlab/ci folder incorporates the required templates to change from the default pipeline machinery jobs. These are also included in the .gitlab-ci.yml file:

# Deployments
- local: "/.gitlab/ci/common.gitlab-ci.yml"
- local: "/.gitlab/ci/deploy-test.gitlab-ci.yml"
- local: "/.gitlab/ci/low-itf.gitlab-ci.yml"
- local: "/.gitlab/ci/mid-itf.gitlab-ci.yml"
- local: "/.gitlab/ci/psi-low.gitlab-ci.yml"
- local: "/.gitlab/ci/psi-mid.gitlab-ci.yml"
- local: "/.gitlab/ci/stfc-techops.gitlab-ci.yml"
- local: "/.gitlab/ci/stfc-dp.gitlab-ci.yml"

Each of this files includes all of the jobs for a particular datacentre. Splitting by datacentre makes it easy to control the variables for each, as there might be slight differences between them.

CI/CD is aligned in such a way that MID and LOW should be handled through configuration options only. All of the deployment, information, test, redeployment and destruction jobs share the same code, regardless of the datacentre, differing only on variable assignments. This makes it more predictable and reliable.

Note

We’ve included the same datacentre-driven separation of the stages into the resources folder, were we can organize datacentre/telescope specific values.yaml files. These are automatically added to each respecting deployment job in the CI/CD pipeline.