Getting started

Contributions to this API are welcome.

A contribution can be either be a patch or a minor/major feature. Patches include bugfixes and small changes to the code. Features can either be minor or major developments and can include potentially disruptive changes.

Development environment

  1. Clone the repository locally

git clone <ska-src-api-global-execution-url>

  1. Initialise submodules for standard make targets and variables

ska-src-api-global-execution$ git submodule update --recursive --init

Development cycle

Makefile targets have been included to facilitate easier and more consistent development. The general recipe is as follows:

  1. Depending on what you are working on, fork the project and create a new major/minor/patch branch, e.g.

    ska-src-api-global-execution$ make patch-branch NAME=some-name

    Note that this both creates and checkouts the branch.

  2. Make your changes.

  3. Create new (OpenAPI) code samples if necessary (requires the service to be running):

    ska-src-api-global-execution$ make code-samples

  4. Set python version in poetry to 3.13

    ska-src-api-global-execution$ poetry env use 3.13

  5. Find the python interpreter executable path

    ska-src-api-global-execution$ poetry env info --executable

  6. Active poetry environment

    ska-src-api-global-execution$ eval $(poetry env activate)

  7. Update poetry.lock

    ska-src-api-global-execution$ poetry lock --no-update

  8. Install dependencies

    ska-src-api-global-execution$ poetry install

  9. Add your changes to the branch:

    ska-src-api-global-execution$ git add ...

  10. Bump the version and commit, entering a commit message when prompted:

ska-src-api-global-execution$ make bump-and-commit ``` This is essential to keep version numbers consistent across the helm chart and python package.

  1. Push the changes to your fork when ready:

ska-src-api-global-execution$ make push ```

  1. Create a merge request against upstream main.

Development tricks

Using poetry

  1. To work inside a poetry shell:

ska-src-api-global-execution$ poetry shell

  1. To install dependencies from pyproject.toml:

(venv)ska-src-api-global-execution$ poetry install

Managing authorisation

Authorisation is managed by dependencies. See the examples router for how to connect to the Permissions API.

Bypassing AuthN/Z

AuthN/Z can be bypassed for development only by setting DISABLE_AUTHENTICATION=yes in the environment. However you need to use a valid JWT token in the header. An expired valid access token will also work

Amending charts

The CI/CD pipeline uses templates inherited from the wider SKA project.

If making changes to the chart, it may be necessary to alter the chart parameters that are passed to both the CI/CD pipeline and local deployment.

Common parameters that are passed to both can be altered in testing.mk under the variable K8S_CHART_COMMON_PARAMS. Variables for the CI/CD and local deployments can be found under the variable K8S_CHART_PARAMS under cicd.mk and dev.mk respectively.

Managing connections to other APIs

See the examples router for how to connect to the Site Capabilities API.

Testing

Testing is done via the pytest module, with code coverage provided by the pytest-cov module.

Component testing

Component testing is conducted inside a k8s deployment environment using mocked responses to external services.

The component tests implemented for this repository are stored under the /tests/component directory. These component tests are executed during the test stage of the CI/CD pipeline under the k8s-test-api-with-disabled-auth and k8s-test-api-with-enabled-auth jobs.

For local testing, an environment can be installed via minikube/helm with:

ska-src-api-global-execution$ minikube start
ska-src-api-global-execution$ make k8s-install-chart

Note that if only tests are modified, it isn’t necessary to run the k8s-install-chart target.

To run the tests locally with both authentication enabled and disabled, respectively:

ska-src-api-global-execution$ make k8s-test-auth
ska-src-api-global-execution$ make k8s-test-noauth

Code quality

This repository uses the following libraries for code quality:

  • isort for sorting imports,

  • black to enforce a consistent coding style,

  • flake8 to check code base against coding style (PEP8), and

  • pylint to look for programming errors and code smells

Linting

Operations for code linting are performed by the python-lint Makefile target provided by the .make submodule. This should be run inside a poetry shell, e.g.

$ poetry shell
(ska_src_api_global_execution-py3.12) ska_src_api_global_execution$ make python-lint

Formatting

Operations for code formatting are performed by the python-format Makefile target provided by the .make submodule. This should be run inside a poetry shell e.g.

$ poetry shell
(ska_src_api_global_execution-py3.12) ska_src_api_global_execution$ make python-format

Generally, before any push hit this command make python-do-format && make python-do-lint

Documentation

There is a Makefile target for generating documentation locally:

ska-src-api-global-execution/docs$ make html

To render inheritance diagrams etc., the graphviz library must be installed.

Fetch tokens

How to site-capabilities or data-management API token

curl --request POST \
  --url https://ska-iam.stfc.ac.uk/token \
  --header 'Content-Type: multipart/form-data' \
  --header 'User-Agent: insomnia/10.2.0' \
  --form grant_type=client_credentials \
  --form client_id=$CLIENT_ID \
  --form client_secret=$CLIENT_SECRET \
  --form audience=site-capabilities-api

For calling Global execution API you need to have a proper user access token with global-execution-api audience.
This is how I am fetching the IAM token which I use to call the GE-API service.

git clone gitlab.com:ska-telescope/src/src-sa/ska-src-sa-cli.git
cd ska-src-sa-cli

poetry lock
poetry install

# To find a new token using IAM device flow
srcnet-oper token request

# Token exchange for global-execution-api
srcnet-oper token exchange global-execution-api

# Find the token present in the disk 
srcnet-oper token ls
/tmp/srcnet/user/f772c41e-9b32-4e27-a7a8-e1fd622e955e.token

# Here you will find the actual token
cat /tmp/srcnet/user/f772c41e-9b32-4e27-a7a8-e1fd622e955e.token | jq

Contributing guidelines

  • Create a new branch for each feature or bug fix. The branch name should be should be either a jira feature id or story id.

  • Update the CHANGELOG.md file with the changes you have made. Follow the format used in the existing changelog entries.

  • Write tests for your changes and ensure that all tests pass before submitting a pull request.

  • Follow the coding standards and best practices used in the project.

  • Ensure that your code is well-documented and includes comments where necessary.

  • Submit a pull request to the main branch and provide a clear description of the changes you have made.

  • Core maintainers will review the pull request and provide feedback. Address any feedback and make necessary changes before the pull request is merged.

  • Core maintainers will update the release, tags and merge the pull request once it has been approved and all checks have passed.

  • Core maintainers will also update the changelog with the new release version and the changes included in that release.

  • For deployment, core maintainers will update the helm chart with the new version and deploy the changes to the appropriate environment (e.g., staging, production).

  • Currently, it is deployed in CHSRC DEV and PROD environment. It will be deployed in staging environment in the near future.

  • We are already following the release management process. However, few things are still pending, and we are planning to address them in this PI.

  • With the gitlab pipeline it is automatically creating PyPI package, helm chart, OCI image and pushing them to the respective repositories.

  • This project is also added to the SonarQube for code quality and security analysis. We are planning to add more quality gates in the near future.

Versioning Standard

All SRCnet software releases to follows Semantic Versioning (SemVer 1.0.0):

  • MAJOR – Breaking or incompatible changes

  • MINOR – Backward-compatible feature additions

  • PATCH – Bug fixes only (no new features)

How to tag a release

git checkout main
git pull origin main
git tag -a 0.1.13 -m "Release version 0.1.13"
git push origin 0.1.13

Coding Standard to follow

Things to Know Before You Start Developing

  • This repo is created with the help of Template Repo created and we are adding new functionalities on top of that, but we do not change the core structure.

  • server.py is the entry point of the application. It initializes the FastAPI app and includes all the routers.

  • Routers are responsible for defining the API endpoints and handling HTTP requests. Each feature should have its own router.

  • Controllers should not contain any business logic. They should only handle the request and response objects and call the appropriate service methods.

  • Services contain the business logic of the application. They should be reusable and independent of the web framework.

  • Dependency Injection: We should not create dependencies directly in any of the layers (routers, controllers, services). Instead, we should use dependency injection to manage dependencies and promote loose coupling.

  • FastAPI’s Dependency Injection System using Depends: Used to inject dependencies into the routers, controllers, and services. This allows for better testability and maintainability of the code.

  • beans.py: It is hard to track the beans and their dependencies in a large application. To solve this problem, we have a dedicated file called beans.py that is responsible for managing the dependencies and their injection.

  • Singleton Pattern: We largely follow singleton pattern for the classes in the application. This means that we create only one instance of a class and reuse it throughout the application. This is achieved using the Singleton metaclass.

  • There are already setup made to call the Site-Capabilities, and it can be extended to use other services like Data-Management-API, Accounting-API etc.

  • **ConfigClient for Configuration Management:**It allows us to segregate configurations based on profiles and automatically resolve environment variables. (A new version of the ConfigClient will be released in the near future)

    • The ConfigClient can automatically resolve environment variables and default values, for example: connect_timeout: ${REQUEST_CONNECT_TIMEOUT:5}.

    • app.yaml is the entry point (all configurations can be added here).

    • Configurations can be split into profiles such as app-dev.yaml and app-prod.yaml.

    • Specific profiles can be loaded using application.config.profiles (e.g., panda, logger, dev).

    • Static configuration files can be added using application.config.load (e.g., /etc/cfg/cfg-1.yaml, /etc/cfg/cfg-2.yaml).

  • Logger Client: Use the logger client as follows and use DEBUG, INFO, ERROR, WARN logs wherever possible. It is possible to package wise different logger level and environment wise logging:

    logger = LoggerClient.get_logger(name)

Steps to add a new endpoint

  1. Create a specific-router.py file for the feature inside src/ska_src_api_global_execution/rest/routers.

  2. Add the router to src/ska_src_api_global_execution/rest/server.py (e.g., app.include_router(jobs_router)).

  3. Create the endpoints in the router (refer to routers/jobs.py as an example).

  4. Create the controller and service classes in the appropriate directories. Use (metaclass=Singleton) to make these classes singletons.

  5. Create the required dependency-injection methods in src/ska_src_api_global_execution/rest/beans.py.

  6. Use ConfigClient for configuration segregation and automatic environment variable resolution.

Code flow

  1. The client sends an HTTP request to the API endpoint defined in the router.

  2. The router receives the request and calls the appropriate controller method.

  3. The controller method processes the request and calls the corresponding service method to perform the business logic.

  4. The service method executes the business logic and returns the result to the controller.

  5. The controller receives the result from the service and constructs the HTTP response.

  6. The router sends the HTTP response back to the client.

High level Architecture

High Level Architecture

POST /worktep API flow