Managing dependencies

This section describes the basic usage of skart and how to set up a repository if one wants to make use of the dependency management functionality.

skart has been developed to ease and improve continuous integration between inter-dependent SKA repositories, which have projects in GitLab and build and release their artefacts using the standard CI templates. It searches specific CI pipeline jobs for artefact names, therefore it is important that the repositories we work with are set up accordingly.

Repository Setup

So what would an inter-repository continuous integration workflow look like? For every repository, we should:

1. Double-check whether we could merge repositories: While skart smoothens things out a bit, it has to be stressed that wherever possible, related code should be kept in the same repository. However, that is clearly not always possible, as sometimes separate artefacts need to be produced.

2. Set up skart.toml: This should list all dependencies that we consider part of the same “mainline”. The logic of continuous integration suggests that we should rather err towards being inclusive here, so the suggestion is to cover all SKA dependencies at minimum. For details of what package types are supported and how to set them up see the Toml File section.

   This should support a “release” mode (see Getting Started), which only selects tagged versions of dependencies. For example:

   [dep.ska-sdp-config]
   dep_type = 'gitlab:python'
   project = 'ska-telescope/sdp/ska-sdp-config'
   release.tag = '\d+\.\d+\.\d+'
   
   [dep.ska-sdp-config.sink.poetry]
   sink_type = 'python:poetry'
   release.version_only = true
   

   This sets up such that for release mode we only select packages with a version tag (a.b.c), and does not pull packages from GitLab.

3. Update the Makefile: if you want to use skart together with the dependency management make targets from SKA CI/CD Makefiles, you will need the following information added to the Makefile of your repository:

   include .make/dependencies.mk
   
   PROJECT_NAME = <gitlab repository slug>
   PROJECT_PATH = ska-telescope/<path to repository>
   ARTEFACT_TYPE = <artefact type: oci or python>
   

   If your artefact is neither an oci image nor a python package, use something that is more relevant, e.g. helm-chart. This will mean, however, that skart will not wait for the pipeline being executed on the given artefact when run with some of the make targets. For a list of supported types see the pipelineWait function in the SKA CI/CD Makefiles repository.

4. Set up CI schedules: skart can be used to regularly test code with updated versions of dependencies. Set up and more details can be found in CI scheduled builds.

Running skart

Once the set up is complete, you can already run skart (once it is installed). The skart CLI is described here: skart Command-Line Interface.

For example, one can update dependencies locally, based on the skart.toml file, by running the following command:

skart update

If you want to update with released versions of dependencies, run:

skart update --mode release

Branches

A special case is changes that span multiple repositories, such as when we need to make a change to a dependency in order to make a feature in another repository work. In development, this can be managed simply through installing working versions into a shared local (virtual) environment, but ordinarily things become much more complicated in GitLab CI: one would have to go through an entire release process in order to make the change in the dependency able to flow downstream.

With skart there is a simpler option: the default branch to find packages for in dependencies defaults to the name of the current branch when executing skart update. This means that when two (or more) repositories have a branch of the same name, skart will prefer packages from those branches over those from the main branch. This means that one can create branches in all relevant repositories and have them test against each other in Gitlab CI.

While this could be a helpful tool for complicated integrations, it has to be stressed that this is not entirely without risk. Clearly if we merge changes into repositories in the wrong order here there is a chance to (temporarily) break the mainline build.

Notes on Python Poetry Dependencies

Poetry dependencies allow us to use the following methods of specifying the version parameters:

Simple Versions:

[tool.poetry.dependencies]
ska-sdp-config = "^1.0.2"

A Version with source parameters:

[tool.poetry.dependencies]
ska-sdp-config = {source = "ska-sdp-config", version = "==1.0.2+dev.cb99a9efa"}

[[tool.poetry.source]]
name = "ska-sdp-config"
url = "https://gitlab.com/api/v4/projects/21141282/packages/pypi/simple"
priority = "supplemental"

A version that comes from a git branch:

[tool.poetry.dependencies]
ska-sdp-dataqueues = { git = "https://gitlab.com/ska-telescope/sdp/ska-sdp-dataqueues.git", branch = "main" }
flask = { git = "https://github.com/pallets/flask.git", rev = "38eb5d3b" }
numpy = { git = "https://github.com/numpy/numpy.git", tag = "v0.13.2" }

Using skart update converts all the above to only having a source and version. Using skart update --release converts all the above to “simple version”. skart only updates dependencies specified in the skart.toml file.

The keys that will be removed if needed are the following: git, branch, rev, tag.

Support for Poetry 1 and Poetry 2

With the introduction of Poetry 2, and the support for the PEP 621 compliant project section, skart has been updated to handle projects that use both purely the tool.poetry sections and those that make use of the project section.

When both [tool.poetry.dependencies] and [project.dependencies] are present, skart updates dependencies in [project.dependencies] with version information whilst preserving information such as the source in the [tool.poetry.dependencies] section. Dependencies originating from [tool.poetry.group.*.dependencies] are preserved and updated within their respective groups.