Continuous Integration in GitLab

Note

Some knowledge of the GitLab CI configuration is assumed

The gitlab-ci.yml CI controller

It is proposed that we add pipelines as stages in the gitlab-ci, thereby utilising the test infrastructure to perform basic pipeline tests.

The CI file is organised in stages:

image: ubuntu:18.04

stages:
    - build-all
    - build-yanda
    - test
    - deploy

before_script:
    - echo "Before script section"
    - echo "For example you might run an update here or install a build dependency"
    - echo "Or perhaps you might print out some debugging details"

after_script:
    - echo "After script section"
    - echo "For example you might do some cleanup here"

build-all:
stage: build-all
script:
    - echo "Common build for everyone"
    - mkdir ${CI_PROJECT_DIR}/products
artifacts:
    paths:
    - ${CI_PROJECT_DIR}/products

build-yanda:
stage: build-yanda
script:
    - echo "Do your build here"
    - mkdir ${CI_PROJECT_DIR}/products/yandasoft
artifacts:
    paths:
    - ${CI_PROJECT_DIR}/products/yandasoft

test-yandasoft:
image: registry.gitlab.com/askapsdp/all_yandasoft
stage: test
script:
    - echo "The YANDASoft test"
    - apt-get update -y
    - apt-get install -y ca-certificates curl --no-install-recommends
    - apt-get install -y default-jdk
    - cd pipelines/yandasoft/nextflow
    - curl -s https://get.nextflow.io | bash
    - ./nextflow run ./functest.nf
    - mv products/* ${CI_PROJECT_DIR}/products/yandasoft/
artifacts:
    paths:
    - ${CI_PROJECT_DIR}/products/yandasoft

test-rascil:
image:
name: $CAR_OCI_REGISTRY_HOST/rascil-full:0.3.0
entrypoint: [""]
stage: test
before_script:
    - echo "The RASCIL test"
    - apt-get update
    - apt-get -y install default-jdk
    - cd pipelines/rascil/nextflow
    - curl -s https://get.nextflow.io | bash
script:
    - ./nextflow run rascil_imager.nf
    - mv products/* ${CI_PROJECT_DIR}/products/rascil/
artifacts:
paths:
    - ${CI_PROJECT_DIR}/products/rascil
expire_in: 1 week

test-other:
stage: test
script:
    - echo "Do another parallel test here"
    - echo "For example run a lint test"

deploy1:
stage: deploy
script:
    - echo "Do your deploy here"

# Create Gitlab CI badges from CI metrics
# https://developer.skatelescope.org/en/latest/tools/continuousintegration.html#automated-collection-of-ci-health-metrics-as-part-of-the-ci-pipeline
include:
- project: 'ska-telescope/templates-repository'
    file: 'gitlab-ci/includes/post_step.yml'

You can see in this example current as of March 2021. I have included a test pipeline for the YANDASoft imager. The test is performed inside a nextflow pipeline - it is proposed that this CI is extended by other pipelines. It is further possible that more tests can be performed upon the products archived by the pipelines.

Note

The NextFlow method is detailed in the YANDASoft pipeline. But this not meant to be prescriptive. This is simply one way of presenting a containerised workflow