SKAMPI Chart Integration

The SKA evolutionary prototype, also known as the Minimum Viable Product (MVP), results from the integration of the several telescope subsystems. This integration is done in the SKA MVP Prototype Integration Deployment Gitlab repository. The tests concerning SKAMPI, are in SKA MVP Prototype Integration Tests Gitlab repository.

If you are developing (or planning to develop or contribute to) a component or subsystem that is to be integrated in SKAMPI, please make sure that you are already familiar with:

• How the SKA does Continuous Integration and Deployment

• The SKA Definition of Done

• The SKA Software Testing Policy and Strategy

We also recommend the SKAO Pipeline Machinery Tutorial for a hands-on overview of present SKAO GitLab CI/CD infrastructure.

Before integrating your component in SKAMPI it is valuable to both understand and try to deploy it along with the SKA Tango Examples.

Before you begin

Before you start with the actual integration you need to clone the SKAMPI repository and test if its deployment is working. When cloning the SKAMPI repository all the relevant submodules need to be imported. We recommend using ssh to clone, but it can also be done using:

$ git clone --recurse-submodules git@gitlab.com:ska-telescope/ska-skampi-deployment.git
$ cd ska-skampi+deployment

If you require a new branch to contribute, make sure you follow the SKA Git guidelines, branching strategy and best practices.

Adding the new or revised chart

Adding a chart for the first time or replacing a chart with a new release is a relatively straightforward process, and should be consistent across the different telescope subsystems. It generally requires very similar changes in the following files:

• charts/ska-low/Chart.yaml - SKA LOW umbrella chart definition

• charts/ska-mid/Chart.yaml - SKA MID umbrella chart definition

• pipeline.yaml - Primary values file passed to the Helm chart installation procedure

All file paths are relative to the root of the repository. The low and mid charts default values (charts/ska-<telescope>/values.yaml) are modified with the values in pipeline.yaml. Note that one can set a custom VALUES=<space separated paths to values files> in order to pass different values, for instance, depending on the environment, or datacentre where SKAMPI will be deployed.

Changing the umbrella charts

The charts/ska-low/Chart.yaml and charts/ska-mid/Chart.yaml files are Helm umbrella charts determining which Helm charts should used in SKA-LOW and SKA-MID deployments. New charts should be added just below dependencies: as shown in the example below for ska-tango-base and ska-tango-util.

 apiVersion: v2
 name: ska-mid
 description: A Helm chart for Kubernetes MVP Mid
 type: application
 version: 0.8.2
 appVersion: 0.8.2
 icon: https://www.skatelescope.org/wp-content/uploads/2016/07/09545_NEW_LOGO_2014.png
 dependencies:
 - name: ska-tango-base
   version: 0.3.5
   repository: https://artefact.skao.int/repository/helm-internal
   condition: ska-tango-base.enabled
- name: ska-tango-util
  version: 0.3.5
  repository: https://artefact.skao.int/repository/helm-internal
(...)
- name: ska-some-new-chart
  ...

The values for the key fields name, version, repository, should be present. Specifying a condition is often preferred, as some components are not required for certain deployments.

Checking chart dependencies

In order to know the the versions of the charts our umbrella charts use, we can run the following target:

make helm-check-deps

This will search the dependency repositories for the latest version of each dependency. With this target, we can understand how out-of-date our umbrella chart dependencies are. Its is important that the versions of the components that get released are updated in SKAMPI, so that it is a representative MVP of the existing software.

Enabling your chart in the pipeline

In the root of your local ska-skampi-deployment folder there is a pipeline.yaml file which looks like this:

minikube: false

# Common products
ska-tango-base:
  vnc:
    enabled: false
  tangodb:
    use_pv: true
ska-sdp:
  enabled: true
  ska-sdp-dataproduct-dashboard:
    enabled: false
ska-oso-scripting:
  enabled: false
ska-taranta:
  enabled: true
ska-ser-skuid:
  enabled: true
ska-tango-archiver:
  enabled: true
ska-landingpage:
  enabled: true

# Mid specific products
ska-mid-cbf:
  legacy_compatibility: true
  enabled: true
ska-csp-lmc-mid:
  legacy_compatibility: true
  enabled: true
ska-tmc-mid:
  legacy_compatibility: true
  enabled: true
ska-sim-dishmaster:
  legacy_compatibility: true
  enabled: true

# Low specific products
ska-tmc-low:
  legacy_compatibility: true
  enabled: true
ska-low-mccs:
  legacy_compatibility: true
  enabled: false

The pipeline.yaml file controls all the variables that are used by Helm when interpreting the templates written for each of the Charts. Make sure that the subsystem that you are adding is enabled in this file and please avoid enabling minikube in it. If you are going to deploy SKAMPI without the CICD pipelines for testing purposes, create a copy of this file - call it *values.yaml - and use that one instead, enabling both your Helm chart and minikube on it. You can then set multiple files as the helm values as follows:

$ echo VALUES=local_values.yaml other_values.yaml >> PrivateRules.mak

For the guidelines specific to minikube you can get all the relevant information from |how to deploy and interact with SKAMPI|.

Tip

If the behaviour of your chart is somewhat unexpected you should also verify the values.yaml file present in the same folder as the Umbrella chart. You may need to change the configuration in that file.

Merge Requests

Even before you have included all relevant changes, preferably as soon as the first commit, you should start a Merge Request on Gitlab so that the review process can start. Please mark it as draft to ensure that no merge can be done even by accident. The merge is only done after the reviewers, drafted from multiple teams, reach a consensus. The ticket can then be marked as ready for acceptance.

The SKAMPI review process is of course based on The SKA Definition of Done but is different in that it should go through the changes done not only in the SKAMPI repository, but also in other repositories that contributed to the subsystem being deployed.

This is required since the story associated with the Merge Request will often be part a feature, and the larger context needs to be considered. It is a lot of work but the stability of SKAMPI absolutely requires it. The whole process is described in more detail in the following section.

The SKAMPI review process

Whenever a team feels that a new chart can be added to SKAMPI it should start the review process by assembling a group of reviewers, including members from other teams, and preparing the tools to capture the conversation. Code reviews, feature discussion, and problem resolution, should be made available not only on the Merge Request page on Gitlab, but also on Confluence pages, a dedicated Slack channel, and linked to Jira.

The goals of the cross-team review are to:

• Ensure that the delivered product meets requirements

• Increase the quality of code and the feature being delivered

• Harmonise and standardise the development practices

• Share a common understanding of the SKA software system and its architecture

• Give developers the opportunity to gain technical insight about all areas of code

• Ensure the test suite is appropriate, and provides adequate coverage for the feature scope in support of acceptance

• Increase the ability to collaborate between teams

Scheduled review meetings should happen in the form of an informal code-walk-through. This can happen in a focused meeting, using tools like zoom and supported by other documentation that can be shared with the reviewers in advance of the meeting. If possible the meeting should be recorded and linked to Jira.

The information to be shared needs to include:

• The Feature, as described on Jira.

• A pointer to relevant documentation describing the subsystem under review. For example this could be part of the Solution Intent, or part of the detailed design documentation in the developer portal.

• The set of Stories realising the features.

• The set of Merge Requests contributing to the feature.

• Relevant tests and related outputs.

During the review the authors will:

• Describe the larger context where the feature takes place with a brief overview of the subsystem.

• Describe the detailed design behind the implementation

• Walk through the code, possibly following the flow of execution

The reviewers will in turn:

• Verify that the default SKA pipeline machinery (templates, makefiles) is used in all relevant repositories

• Ask questions about the design choices

• Verify that the implementation adheres to SKA quality standards

• Verify that the feature is supported by tests