SKA CI/CD Templates Repository¶
This repository holds the common job templates used by the pipelines in the SKA Projects.
It is developed to provide common functionality across SKAO CI/CD by calling the make targets residing in ska-cicd-makefile repository.
For any help requests, suggestions, improvements or any issues you may have with the content of this repository please drop us a message on #team-system-support channel at SKAO slack.
Disclaimer¶
The latest releases of ansible (v7.1.0 and v7.0.0) introduced a bug into ansible-core (v2.14.1) that breaks ansible-lint. In order to fix this the ansible version being installed is going to be pinned to version 6.7.0 and should go back to the latest version once the bug has been patched. Ansible-lint version has also been temporarily pinned to version 6.11.0
Templates¶
Currently the following artefact types are supported:
- ansible: ansible collections
- helm: Helm Charts
- oci-images: OCI Images
- python: Python Code and Artefacts
- raw: Raw Artefacts
- conan: Conan Artefacts
- rpm: RPM Artefacts
- gpu: Python Code and Artefacts with GPU support for the testing stage
- terraform: Terraform Code
In addition, several use-cases for a typical repository are also supported:
- docs: Sphinx documentation support for ReadTheDocs
- k8s: Kubernetes in-cluster testing
- release: GitLab tag based release and changelog automation
- finaliser: GitLab CI Badges and Pipeline workflow rules
- tmdata: Upload telescope model data
- xray: upload BDD test results to XRAY
Template Inheritance¶
There is a hierarchy to the templates which express a certain amount of inheritance, through nested includes:
graph LR
NodeRoot[Template Hierarchy] --> NodeFinaliser[finaliser.gitlab-ci.yml]
NodeRoot[Template Hierarchy] --> NodeRelease[release.gitlab-ci.yml]
NodeRoot[Template Hierarchy] --> NodeDocs[docs.gitlab-ci.yml]
NodeRoot[Template Hierarchy] --> NodeRaw[raw.gitlab-ci.yml]
NodeRoot[Template Hierarchy] --> NodeRPM[rpm.gitlab-ci.yml]
NodeRoot[Template Hierarchy] --> NodeConan[conan.gitlab-ci.yml]
NodeRoot[Template Hierarchy] --> NodeAnsible[ansible.gitlab-ci.yml]
NodeRoot[Template Hierarchy] --> NodeHelm[helm-chart.gitlab-ci.yml]
NodeRoot[Template Hierarchy] --> NodeK8s[k8s.gitlab-ci.yml]
NodeRoot[Template Hierarchy] --> NodeK8sTestRunner[k8s-test-runner.gitlab-ci.yml]
NodeRoot[Template Hierarchy] --> NodeDeploy[deploy.gitlab-ci.yml]
NodeRoot[Template Hierarchy] --> NodeOCI[oci-image.gitlab-ci.yml]
NodeRoot[Template Hierarchy] --> NodePython[python.gitlab-ci.yml]
NodeRoot[Template Hierarchy] --> NodeTerraform[terraform.gitlab-ci.yml]
NodeRoot[Template Hierarchy] --> NodeTMData[tmdata.gitlab-ci.yml]
NodeRoot[Template Hierarchy] --> NodeXray[xray-publish.gitlab-ci.yml]
NodeRoot[Template Hierarchy] --> NodeConfigCapture[config-capture.gitlab-ci.yml]
subgraph "Finaliser/Badge Templates"
NodeFinaliser --> NodeFinaliserRules[rules.gitlab-ci.yml]
end
subgraph "Release Management Templates"
NodeRelease --> NodeReleaseChangeLogBuild[changelog-build.gitlab-ci.yml]
end
subgraph "Docs Templates"
NodeDocs --> NodeDocsBuild[docs-build.gitlab-ci.yml]
NodeDocs --> NodeDocsPages[docs-pages.gitlab-ci.yml]
end
subgraph "Raw Package Templates"
NodeRaw --> NodeRawBuild[raw-build.gitlab-ci.yml]
NodeRaw --> NodeRawPublish[raw-publish.gitlab-ci.yml]
end
subgraph "RPM Package Templates"
NodeRPM --> NodeRPMBuild[rpm-build.gitlab-ci.yml]
NodeRPM --> NodeRPMPublish[rpm-publish.gitlab-ci.yml]
end
subgraph "Conan Package Templates"
NodeConan --> NodeConanBuild[conan-build.gitlab-ci.yml]
NodeConan --> NodeConanPublish[conan-publish.gitlab-ci.yml]
end
subgraph "Ansible Templates"
NodeAnsible --> NodeAnsibleLint[ansible-lint.gitlab-ci.yml]
NodeAnsible --> NodeAnsiblePublish[ansible-publish.gitlab-ci.yml]
end
subgraph "Helm Templates"
NodeHelm --> NodeHelmLint[helm-chart-lint.gitlab-ci.yml]
NodeHelm --> NodeHelmBuild[helm-chart-build.gitlab-ci.yml]
NodeHelm --> NodeHelmPublish[helm-chart-publish.gitlab-ci.yml]
end
subgraph "Kubernetes Templates"
NodeK8s --> NodeK8sBuild[k8s-test.gitlab-ci.yml]
NodeK8sTestRunner
end
subgraph "Xray Templates"
NodeXray
end
subgraph "Config Capture Templates"
NodeConfigCapture
end
subgraph "Deployment Templates"
NodeDeploy --> NodeDeployDev[deploy.dev.gitlab-ci.yml]
NodeDeploy --> NodeDeploySharedDev[deploy.shared.dev.gitlab-ci.yml]
NodeDeploy --> NodeDeployIntegration[deploy.integration.gitlab-ci.yml]
NodeDeploy --> NodeDeployStaging[deploy.staging.gitlab-ci.yml]
end
subgraph "OCI Image Templates"
NodeOCI --> NodeOCILint[oci-image-lint.gitlab-ci.yml]
NodeOCI --> NodeOCIBuild[oci-image-build.gitlab-ci.yml]
NodeOCI --> NodeOCIPublish[oci-image-publish.gitlab-ci.yml]
NodeOCI --> NodeOCIScan[oci-image-scan.gitlab-ci.yml]
end
subgraph "Python Templates"
NodePython --> NodePythonLint[python-lint.gitlab-ci.yml]
NodePython --> NodePythonBuild[python-build.gitlab-ci.yml]
NodePython --> NodePythonTest[python-test.gitlab-ci.yml]
NodePython --> NodePythonPublish[python-publish.gitlab-ci.yml]
end
subgraph "Terraform Templates"
NodeTerraform --> NodeTerraformLint[terraform-lint.gitlab-ci.yml]
end
subgraph "Telescope Model Data Templates"
NodeTMData --> NodeTMDataBuild[tmdata-package.gitlab-ci.yml]
NodeTMData --> NodeTMDataPublish[tmdata-publish.gitlab-ci.yml]
end
To learn more about how to use the templates, follow How to Use section.
To learn more about how the templates are designed and integrated, follow Design section.
Pipeline Rules¶
The finaliser.gitlab-ci.yml
file includes top-level workflow rules(rules.gitlab-ci.yml
) that enables the Merge Request Pipelines and prevents duplication with the Branch Pipelines. Please see below where the different types of pipelines are described. It also enables the use of Merged Result pipelines.
Branch Pipelines: pipelines that target a source branch, i.e. the pipeline is run against what you pushed to the remote every time.
Merge Request Pipelines: pipelines that are attached to a merge request. They are different because they only run against commits associated with a merge requests
rules.gitlab-ci.yml
file(workflow:rules
) defines whether the pipeline should be created or not at all.
This is to manage branch and merge request pipelines while preventing duplicate pipelines. For example, without any rules if there is an open MR, there would be two identical pipelines running, one for the MR pipeline and the other for the branch pipeline. The added workflow:rules
prevents this by enabling/disabling one of the pipeline types.
The current workflow is if there are no open merge requests, a branch pipeline is created. If there is an open merge request, then the branch pipelines are disabled and merge request pipelines are enabled. So, there would only be single pipeline in the end.
The above is needed for the merged result pipelines. It enables the pipeline to really see the results of the master branch pipeline as if the changes are merged instead of just seeing the results for the source branch.
To illustrate, It can easily happen that a new commit in a branch works (i.e. the pipeline status is green, and as a result the Merge Request seems safe to merge), while in fact the merge breaks the pipeline in the master branch. By using Merge Result pipelines we can potentially avoid this, by running pipelines on what the master branch would like if our changes were merged. This setting is enabled for all projects in SKAO group. To learn more, please have a look at the GitLab blog post about it.
There are some conditions:
- The MR has to be ready, i.e. no Draft or WIP tags in the MR title
- There shouldn’t be any merge conflicts If the MR is not mergeable (above conditions), then what we have is the old pipelines that are targeting the source branch as usual. However, these are now labelled as detached by GitLab (I think poor choice of words) to differentiate them from merged result pipelines. Fear not! They are still targeting your branch so you are not actually in a detached state from your branch.
How to Use¶
First, add the ska-cicd-makefile repository as a submodule to your project.
git submodule add https://gitlab.com/ska-telescope/sdi/ska-cicd-makefile.git .make
Make sure that you add and commit the .gitmodules
file and .make
directory:
git add .gitmodules .make
git commit -s
Now include the *.mk
files in your Makefile
:
include .make/*.mk
Or, you can include specific ones like below. Note that you have to include the ones you are going to use in your pipeline.
# include makefile targets for Kubernetes management
-include .make/k8s.mk
## The following should be standard includes
# include core makefile targets for release management
-include .make/base.mk
# include your own private variables for custom deployment configuration
-include PrivateRules.mak
To ensure that your GitLab CI pipeline automatically clones submodules, add the following to .gitlab-ci.yml
:
variables:
GIT_SUBMODULE_STRATEGY: recursive
In most scenarios, include the appropriate main template files in your repositories gitlab-ci.yml
file. To include python, oci, kubernetes and documentation support:
include:
# Python
- project: 'ska-telescope/templates-repository'
file: 'gitlab-ci/includes/python.gitlab-ci.yml'
# Terraform
- project: 'ska-telescope/templates-repository'
file: 'gitlab-ci/includes/terraform.gitlab-ci.yml'
# Docs pages
- project: 'ska-telescope/templates-repository'
file: 'gitlab-ci/includes/docs-pages.gitlab-ci.yml'
# k8s steps
- project: 'ska-telescope/templates-repository'
file: 'gitlab-ci/includes/k8s.gitlab-ci.yml'
# configuration capture
- project: 'ska-telescope/templates-repository'
file: 'gitlab-ci/includes/config-capture.gitlab-ci.yml'
# OCI Images
- project: 'ska-telescope/templates-repository'
file: 'gitlab-ci/includes/oci-image.gitlab-ci.yml'
# Raw
- project: 'ska-telescope/templates-repository'
file: 'gitlab-ci/includes/raw.gitlab-ci.yml'
# RPM
- project: 'ska-telescope/templates-repository'
file: 'gitlab-ci/includes/rpm.gitlab-ci.yml'
# Conan
- project: 'ska-telescope/templates-repository'
file: 'gitlab-ci/includes/conan.gitlab-ci.yml'
# Tag Based GitLab Release Management
- project: 'ska-telescope/templates-repository'
file: 'gitlab-ci/includes/release.gitlab-ci.yml'
# .post step finalisers eg: badges
- project: 'ska-telescope/templates-repository'
file: 'gitlab-ci/includes/finaliser.gitlab-ci.yml'
Note, finaliser.gitlab-ci.yml
is required by every repository as it includes functionality for ci-badges and workflow rules which is explained in Templates section.
If you don’t require every job for an artefact in your project, you could also just include the necessary part:
include:
# We don't need python build and publish
# as we are not releasing a python artefact
- project: 'skatelescope/templates-repository'
file: 'gitlab-ci/includes/python-lint.gitlab-ci.yml'
- project: 'skatelescope/templates-repository'
file: 'gitlab-ci/includes/python-test.gitlab-ci.yml'
If you want to get detailed help on what each job is going to do, run make long-help
to get additional documentation on the actual make targets.
Customisation¶
All of the jobs should be configured by their respective make-targets and variables and hooks provided. Follow ska-cicd-makefile to learn more.
Example; If you want to include additional pytest
flags, you can define PYTHON_VARS_AFTER_PYTEST
variable in your main Makefile as below:
PYTHON_VARS_AFTER_PYTEST = -m 'not post_deployment' --forked \
--disable-pytest-warnings
Note that although you can achieve the same effect for the pipelines by providing variables in your gitlab-ci.yml
file as below, it is not recommended! as it makes the local development and pipeline behaviour diverge from each other.
# NOT RECOMMENDED!
variables:
- PYTHON_VARS_AFTER_PYTEST: "-m 'not post_deployment' --forked \
--disable-pytest-warnings"
Design¶
For every artefact type in SKAO, there should be a template that addresses some of the common steps in DevSecOps Lifecycle:
- Lint
- Build
- Test
- Package
- Release
- Deploy
Each artefact type has its own template file in <artefact-type>.gitlab-ci.yml
which includes avaiable steps. Each step have its own <artefact-type>-<step>.gitlab-ci.yml
file to describe when and how it should run.
For python;
│ python.gitlab-ci.yml
├─ python-lint.gitlab-ci.yml
├─ python-build.gitlab-ci.yml
├─ python-test.gitlab-ci.yml
├─ python-publish.gitlab-ci.yml
You can check the contents of the python.gitlab-ci.yml
file and investigate the actual step template files under gitlab-ci/includes directory.
For each job, script
part should just compose of calling the make targets associated with the artefact and step. before-script
part provides checking if the target exists and provides documentation.
Providing that the underlying implementation is the same as makefile targets allows repositories and developers to be flexible in their own implementation and aligns both the development and pipeline environment. You can read more on how to customise the make targets in makefile repository documentation
rules
part describes when the job should be included in the pipeline to eliminate unnecessary/invalid jobs in the pipeline. To push artefacts to GitLab or CAR from the publish stage jobs, the git tags are used as identifier.
General Layout¶
For any artefact type:
- the main
<artefact>.gitlab-ci.yml
file has only the consisting include jobs. - Each lifecycle step in a pipeline is named with the correct prefix and should reside in its own
<artefact-type>-<step>.gitlab-ci.yml
file. - Each step template file includes the necessary jobs starting with the step name
<artefact-type>-<step>...
. Multiple jobs are provided to handle different use-cases. i.e. to handle python modern/legacy build system, to manage development and production publishing of artefacts etc. - Each job performs any preliminary actions in
before_script
such as logins, config settings for the environment. - Each job just calls the respective
make targets
and passes any variables that needs to be used inscript
part. The script itself should not have any logic to not deviate from the local development workflow.
If we follow the python example as described above. The main python.gitlab-ci.yml
file only includes other jobs:
# unbrella include for all Python life cycle stages
include:
# Linting stage
- local: gitlab-ci/includes/python-lint.gitlab-ci.yml
# Build stage
- local: gitlab-ci/includes/python-build.gitlab-ci.yml
# Test stage
- local: gitlab-ci/includes/python-test.gitlab-ci.yml
# Publish stage
- local: gitlab-ci/includes/python-publish.gitlab-ci.yml
and the python-publish.gitlab-ci.yml
describes the actual python build stage jobs. As you can see the necessary variables are passed to the make python-publish
target and rules
is used to separate tag pipelines from development pipelines to publish to central artefact repository or gitlab registry, respectively.
# Python publish stage template
python-publish-to-car:
stage: publish
tags:
- k8srunner
image: $SKA_K8S_TOOLS_BUILD_DEPLOY
before_script:
- '[ -f .make/python.mk ] || (echo "File python.mk not included in Makefile; exit 1")'
- 'make help | grep python-publish'
script:
- make PYTHON_PUBLISH_USERNAME=${CAR_PYPI_USERNAME} PYTHON_PUBLISH_PASSWORD=${CAR_PYPI_PASSWORD} PYTHON_PUBLISH_URL=${CAR_PYPI_REPOSITORY_URL} python-publish
rules:
- if: '$CI_COMMIT_TAG'
exists:
- pyproject.toml
- setup.py
python-publish-to-gitlab:
stage: publish
tags:
- k8srunner
image: $SKA_K8S_TOOLS_BUILD_DEPLOY
allow_failure: true
before_script:
- '[ -f .make/python.mk ] || (echo "File python.mk not included in Makefile; exit 1")'
- 'make help | grep python-publish'
script:
- make PYTHON_PUBLISH_USERNAME=gitlab-ci-token PYTHON_PUBLISH_PASSWORD=${CI_JOB_TOKEN} PYTHON_PUBLISH_URL=https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/packages/pypi python-publish
rules:
- exists:
- pyproject.toml
- setup.py