Script Development

The steps to develop and test an SDP script are as follows:

1. Create the script

Clone the ska-sdp-script repository from GitLab and create a new branch for your work.

Create a directory for your script in src:

$ mkdir src/<my-script>
$ cd src/<my-script>

Write the script (<my-script>.py). See the existing scripts for examples of how to do this. The examples Real-time script (Test Real-Time Script) and Batch script (Test Batch Script) are the best place to start. These are meant to give you a general idea of the structure real-time and batch scripts should have, and help develop your own.

List of available Helm charts, which can be used for scripts, and their documentation can be found at SDP Helm Deployer Charts

We use poetry as Python packaging and dependency management software. The related documentation can be found at Poetry. In the development, please create a pyproject.toml file with your script’s Python requirements, e.g.

[tool.poetry]
name = "ska-sdp-script-<my-script>"
version = "0.0.0"
description = "SKA SDP <my-script> Script"
authors = ["SKA SDP Developers"]
license = "BSD-3-Clause"
repository = "<link to script directory in repository>"
documentation = "<link to script documentation>"
package-mode = false

[[tool.poetry.source]]
name = "PyPI"
priority = "primary"

[[tool.poetry.source]]
name = "skao"
url = "https://artefact.skao.int/repository/pypi-internal/simple"
priority = "primary"

[tool.poetry.dependencies]
python = "^3.10"
ska-ser-logging = "^0.4.3"
ska-sdp-scripting = "~0.7.1"

In general, modifying [tool.poetry.dependencies] is enough to add your script’s dependencies.

Create a Dockerfile for building the script image, e.g.

FROM python:3.10-slim

WORKDIR /app

COPY pyproject.toml poetry.lock ./

RUN pip install poetry && \
    poetry config cache-dir /app/.cache && \
    poetry config virtualenvs.create false

RUN poetry install && \
    poetry -n cache clear --all . && \
    rm -rf /app/.cache

COPY <my-script>.py ./
ENTRYPOINT ["python", "<my-script>.py"]

Use the base image of your choice, preferably the latest numbered slim version of it, e.g. python:3.10-slim.

Create a file called version.txt containing the semantic version number of the script.

Create a Makefile containing

NAME := ska-sdp-script-<my-script>
VERSION := $(shell cat version.txt)

include ../../make/Makefile

2. Test the script locally

Build the script image. If you are using minikube to deploy the SDP, run:
```
$ eval $(minikube -p minikube docker-env)
$ make build
```
else, just run the make build command. This will add the image to your minikube or local Docker daemon where it can be used for testing with a local deployment of the SDP.
Deploy SDP locally and start a shell in the console pod.
Add the new script to the configuration DB. This will tell the SDP where to find the Docker image to run the script:
```
ska-sdp create script <kind>:<name>:<version> '{"image": "<docker-image:version>"}'
```
where the values are:
- <kind>: batch or realtime, depending on the kind of script
- <name>: name of your script
- <version>: version of your script
- <docker-image:version>: Docker image you just built from your script, including its version tag.
If you have multiple scripts to add, you can import the definitions with:
```
ska-sdp import scripts my-scripts.yaml
```
An example file for importing scripts can be found at: Example Script Definitions
To run the script, create a processing block, either via the Tango interface, or by creating it directly in the configuration DB with ska-sdp create pb.

3. Finish development and deploy the script

Once you are happy with the script, add it to the GitLab CI file (.gitlab-ci.yml) in the root of the repository. You need to add jobs to publish the development and release versions:
```
publish-dev-<my-script>:
  extends: .publish-dev
  variables:
    SRCDIR: src/<my-script>

publish-release-<my-script>:
  extends: .publish-release
  variables:
    SRCDIR: src/<my-script>
```
This will enable the Docker image to be built and pushed to the SKA artefact repository when it is merged into the master branch.
Add the script to the script definition file scripts.yaml in the root of the repository. By default the SDP uses this file to populate the script definitions in the configuration DB when it starts up.
Create a README.md and add the description and instructions to run your script. Include it in the documentation:
- create a new file in docs/src/scripts/<my-script>.rst
- add the following to it:
.. mdinclude:: ../../../src/<my-script>/README.md
- update docs/src/index.rst
Commit the changes to your branch and push to GitLab.

Script Generator

To speed things up for the developer, a Python script has been developed to automatically generate the source files.

The script will generate all the required source files (Python script, requirements, version number, Dockerfile, Makefile, README) for both real-time and batch scripts.

Note: you will need docopt to be installed for the script to run:

pip install docopt

To run the script:

cd templates
python create_script.py <kind> <name>

For example:

python create_script.py realtime my-realtime

Usage

Create source files for real-time or batch script.

Usage:
    create_script.py <kind> <name>
    create_script.py (-h|--help)

Arguments:
    <kind>    Kind of script (realtime or batch)
    <name>    Name of the script to be created