General guidelines

The SDP Integration repository integrates the components of the SKA Science Data Processor (SDP).

It is a public GitLab project accessible from:

https://gitlab.com/ska-telescope/sdp/ska-sdp-integration

Cloning the repository

You may clone it via SSH or HTTPS. For SSH:

$ git clone git@gitlab.com:ska-telescope/sdp/ska-sdp-integration.git --recurse-submodules

And for HTTPS:

$ git clone https://gitlab.com/ska-telescope/sdp/ska-sdp-integration.git --recurse-submodules

The --recurse-submodules option makes sure that you clone the ska-cicd-makefile submodule together with the repository, which is an important part for development.

Note

Make sure you include --recurse-submodules when you run git pull as well, to keep up-to-date with the submodule too.

Branch-based development

We follow a branch-based development process. New code is always added to a branch created from master. Then a (draft) Merge Request is created, which needs to be reviewed and approved by CODEOWNERS before it can be merged. Only maintainers of the repository can merge code to master. Only code on the master branch is released.

Using poetry

Although the repository does not contain a Python package (its main artefact is the Helm chart), its tests are written in Python. The project uses Poetry for package management. Make sure you have it installed and configured to your needs. This is needed if you are developing and running tests.

Python dependencies of the project are collected in pyproject.toml. A poetry.lock file is also included, which pins the versions of dependencies the code has been tested with. If you change the requirements in pyproject.toml make sure you also update the poetry.lock file.

The dependencies are organised into four groups in the pyproject.toml file:

  • test for running the tests

  • test-dev for developing the tests (formatting and linting utilities)

  • docs for producing the documentation

Repository structure

The main layout of the repository is as follows.

The most important top level files include:

  • .gitlab-ci.yml

  • .readthedocs.yaml

  • CHANGELOG.rst,

  • CODEOWNERS

  • CONTRIBUTORS

  • LICENCE

  • Makefile

  • README.rst

  • poetry.lock

  • pyproject.toml

When developing the code, update these as needed.

Directory structure:

Folder

Description

.gitlab/ci

Contains YAML files for GitLab CI/CD job definitions

.make

Contains the ska-cicd-makefile submodule (not directly changeable from the SDP repository)

charts

Helm charts. The ska-sdp chart is the main product of the repository. Includes data like EDA configuration and Taranta dashboard files.

docs

Documentation and related content

resources/values

Values files that can be used to customise the SDP installation

tests

All of the tests and test-related infrastructure

Documentation

We document using Sphinx and Read The Docs. Every time you change the code in the repository, make sure you review the documentation, and update it as needed.