CI scheduled builds

skart can be used as a tool to check whether a repository continues to pass its unit tests when dependent components such as Python packages, OCI image builds, or dependent chart releases are updated. Either released or development versions of these components can be selected for this testing. This gives early indication of any issues which may need to be resolved before these components are released or developed further.

The Toml File section describes how to set up skart to work in a given repository. This is also prerequisite to run scheduled updates with skart.

It is often useful to automatically run a Gitlab scheduled CI pipeline which uses skart to update the components used before continuing to its normal build and test stages. This GitLab scheduled pipeline can be run at appropriate time(s) of day (often overnight) and with a suitable CI variable set to trigger the skart stage.

Note that this will require all Gitlab repositories referenced in skart.toml to be configured such that they allow access by the scheduled CI jobs. To ensure this, update “Job token permissions” in the “CI/CD” settings of the referenced repository so that they allow access from the repository you are adding the scheduled run to.

As a working example this is the start of the file .gitlab-ci.yml from the repository ska-sdp-proccontrol:

skart-update:
  stage: .pre
  tags:
    - ska-default
  rules:
    - if: '$DO_SKART_UPDATE'
      when:  on_success
    - when: never
  before_script:
    - pip3 install --upgrade --index-url=https://artefact.skao.int/repository/pypi-internal/simple skart
    - poetry config virtualenvs.create false
  script:
    - make deps-update
  artifacts:
    paths:
      - poetry.lock
      - pyproject.toml

This specific example uses the deps-update make target from SKA CI/CD Makefiles, which executes skart update followed by poetry update --lock. Alternatively, one can use the deps-update-release make target to use skart to update with released versions of dependencies.

It can be run on schedule, but also as part of any pipeline, in this specific case, if the DO_SKART_UPDATE CI variable is set. After this .pre stage, the CI pipeline continues as normal through its stages (including unit testing) against any updated artefacts.

In the artifacts/paths section of the CI job, we save files that have been updated by skart and/or poetry to be used in subsequent pipeline steps. It is important you pass the correct files:

  • if your package uses python dependencies, pass the poetry.lock and pyproject.toml files.

  • if your repository builds a helm chart, pass the values.yaml and Chart.yaml files

To make sure that a project gets tested against updated dependencies at least once a day, create a pipeline schedule in GitLab that builds the project with the skart-update job once every night.

Connecting dependent repositories in CI

skart is able to find development versions of dependencies from latest builds of given branches. We can make use of this property and set up a chain of builds of several dependent repositories so that each one of them is built with its most recent dependencies and these most recent builds are then used further in other, dependent repositories.

For example, we have the following dependency tree:

ska-sdp-integration -> ska-sdp-proccontrol  -> ska-sdp-config
                    -> ska-sdp-helmdeploy   -> ska-sdp-config

In this example, ska-sdp-integration depends on two componentes, both of which depend on ska-sdp-config.

We can set up scheduled updates of all four of these repositories using the deps-update make target (since that also updates with poetry). If we use this make target, it is OK if one or more of the dependent repositories do not have a skart.toml file because they don’t depend on any SKA packages. In such cases, poetry will still run and update the lock file (if exists). Set them up in the following order (times are purely for demonstration purposes, you can shift them as needed, but the order needs to stay):

  • start a build of ska-sdp-config at 2:00 am, running make deps-update in the .pre stage.

  • the pipeline using the same set up for ska-sdp-helmdeploy and ska-sdp-proccontrol needs to run about 30 minutes later, e.g. at 2:30 am. This will make sure the pipeline on ska-sdp-config already finished, and skart on the other two repositories will use the artefact built as part of the 2:00 am-schedule, already with updated dependencies.

  • finally, run the scheduled, skart-based pipeline for ska-sdp-integration at 3:00 am, to give time for ska-sdp-proccontrol and ska-sdp-helmdeploy pipeliens to finish, so that skart can update using these most recently built versions of the two components.

If for any reason the given pipelines take longer to finish than the delay in setting up the schedules, skart will wait for any pipelines that are currently running, to finish, before moving on with the update.

Warning

Note however, that if any of the pipelines fail, skart will skip the build of the failed pipeline, and move onto the previous, successful one. This will result in using a dependency that is in fact NOT built with the latest code.