.. _basic_guide: Managing dependencies ===================== This section describes the basic usage of ``skart`` and how to set up a repository if one wants to make use of the dependency management functionality. ``skart`` has been developed to ease and improve continuous integration between inter-dependent SKA repositories, which have projects in GitLab and build and release their artefacts using the `standard CI templates `_. It searches specific CI pipeline jobs for artefact names, therefore it is important that the repositories we work with are set up accordingly. Repository Setup ---------------- So what would an inter-repository continuous integration workflow look like? For every repository, we should: 1. **Double-check whether we could merge repositories:** While ``skart`` smoothens things out a bit, it has to be stressed that wherever possible, related code should be kept in the same repository. However, that is clearly not always possible, as sometimes separate artefacts need to be produced. 2. **Set up** ``skart.toml``: This should list all dependencies that we consider part of the same "mainline". The logic of continuous integration suggests that we should rather err towards being inclusive here, so the suggestion is to cover all SKA dependencies at minimum. For details of what package types are supported and how to set them up see the :ref:`skart_file` section. This should support a "release" mode (see :ref:`Getting Started `), which only selects tagged versions of dependencies. For example: .. code:: toml [dep.ska-sdp-config] dep_type = 'gitlab:python' project = 'ska-telescope/sdp/ska-sdp-config' release.tag = '\d+\.\d+\.\d+' [dep.ska-sdp-config.sink.poetry] sink_type = 'python:poetry' release.version_only = true This sets up such that for ``release`` mode we only select packages with a version tag (``a.b.c``), and does not pull packages from GitLab. 3. **Update the Makefile**: if you want to use ``skart`` together with the dependency management make targets from `SKA CI/CD Makefiles `_, you will need the following information added to the ``Makefile`` of your repository: .. code:: include .make/dependencies.mk PROJECT_NAME = PROJECT_PATH = ska-telescope/ ARTEFACT_TYPE = If your artefact is neither an oci image nor a python package, use something that is more relevant, e.g. helm-chart. This will mean, however, that ``skart`` will not wait for the pipeline being executed on the given artefact when run with some of the make targets. For a list of supported types see the `pipelineWait function `_ in the `SKA CI/CD Makefiles `_ repository. 4. **Set up CI schedules**: ``skart`` can be used to regularly test code with updated versions of dependencies. Set up and more details can be found in :ref:`scheduled_tests`. Running ``skart`` ----------------- Once the set up is complete, you can already run ``skart`` (once it is installed). The ``skart`` CLI is described here: :ref:`cli`. For example, one can update dependencies locally, based on the ``skart.toml`` file, by running the following command: .. code-block:: bash skart update If you want to update with released versions of dependencies, run: .. code-block:: bash skart update --mode release Branches -------- A special case is changes that span multiple repositories, such as when we need to make a change to a dependency in order to make a feature in another repository work. In development, this can be managed simply through installing working versions into a shared local (virtual) environment, but ordinarily things become much more complicated in GitLab CI: one would have to go through an entire release process in order to make the change in the dependency able to flow downstream. With ``skart`` there is a simpler option: the default ``branch`` to find packages for in dependencies defaults to the name of the current branch when executing ``skart update``. This means that when two (or more) repositories have a branch of the same name, ``skart`` will prefer packages from those branches over those from the main branch. This means that one can create branches in all relevant repositories and have them test against each other in Gitlab CI. While this could be a helpful tool for complicated integrations, it has to be stressed that this is not entirely without risk. Clearly if we merge changes into repositories in the wrong order here there is a chance to (temporarily) break the mainline build. Notes on Python Poetry Dependencies ----------------------------------- Poetry dependencies allow us to use the following methods of specifying the version parameters: Simple Versions: .. code-block:: toml [tool.poetry.dependencies] ska-sdp-config = "^1.0.2" A Version with source parameters: .. code-block:: toml [tool.poetry.dependencies] ska-sdp-config = {source = "ska-sdp-config", version = "==1.0.2+dev.cb99a9efa"} [[tool.poetry.source]] name = "ska-sdp-config" url = "https://gitlab.com/api/v4/projects/21141282/packages/pypi/simple" priority = "supplemental" A version that comes from a git branch: .. code-block:: toml [tool.poetry.dependencies] ska-sdp-dataqueues = { git = "https://gitlab.com/ska-telescope/sdp/ska-sdp-dataqueues.git", branch = "main" } flask = { git = "https://github.com/pallets/flask.git", rev = "38eb5d3b" } numpy = { git = "https://github.com/numpy/numpy.git", tag = "v0.13.2" } Using ``skart update`` converts all the above to only having a ``source`` and ``version``. Using ``skart update --release`` converts all the above to "simple version". ``skart`` only updates dependencies specified in the ``skart.toml`` file. The keys that will be removed if needed are the following: ``git``, ``branch``, ``rev``, ``tag``. Support for Poetry 1 and Poetry 2 --------------------------------- With the introduction of Poetry 2, and the support for the PEP 621 compliant ``project`` section, ``skart`` has been updated to handle projects that use both purely the ``tool.poetry`` sections and those that make use of the ``project`` section. When both ``[tool.poetry.dependencies]`` and ``[project.dependencies]`` are present, skart updates dependencies in ``[project.dependencies]`` with version information whilst preserving information such as the source in the ``[tool.poetry.dependencies]`` section. Dependencies originating from ``[tool.poetry.group.*.dependencies]`` are preserved and updated within their respective groups.