Script Development

The steps to develop and test an SDP script are as follows:

1. Create the script

  • Clone the ska-sdp-script repository from GitLab and create a new branch for your work.

  • Create a directory for your script in src:

    $ mkdir src/<my-script>
    $ cd src/<my-script>
    
  • Write the script (<my-script>.py). See the existing scripts for examples of how to do this. The examples Real-time script (Test Real-Time Script) and Batch script (Test Batch Script) are the best place to start. These are meant to give you a general idea of the structure real-time and batch scripts should have, and help develop your own.

    List of available Helm charts, which can be used for scripts, and their documentation can be found at SDP Helm Deployer Charts

  • Create a requirements.txt file with your script’s Python requirements, e.g.

    --index-url https://artefact.skao.int/repository/pypi-internal/simple
    ska-ser-logging
    ska-sdp-scripting
    
  • Create a Dockerfile for building the script image, e.g.

    FROM python:3.10-slim
    
    COPY requirements.txt ./
    RUN pip install -r requirements.txt
    
    WORKDIR /app
    COPY <my-script>.py ./
    ENTRYPOINT ["python", "<my-script>.py"]
    

    Use the base image of your choice, preferably the latest numbered slim version of it, e.g. python:3.10-slim.

  • Create a file called version.txt containing the semantic version number of the script.

  • Create a Makefile containing

    NAME := ska-sdp-script-<my-script>
    VERSION := $(shell cat version.txt)
    
    include ../../make/Makefile
    

2. Test the script locally

  • Build the script image. If you are using minikube to deploy the SDP, run:

    $ eval $(minikube -p minikube docker-env)
    $ make build
    

    else, just run the make build command. This will add the image to your minikube or local Docker daemon where it can be used for testing with a local deployment of the SDP.

  • Deploy SDP locally and start a shell in the console pod.

  • Add the new script to the configuration DB. This will tell the SDP where to find the Docker image to run the script:

    ska-sdp create script <kind>:<name>:<version> '{"image": "<docker-image:version>"}'
    

    where the values are:

    • <kind>: batch or realtime, depending on the kind of script

    • <name>: name of your script

    • <version>: version of your script

    • <docker-image:version>: Docker image you just built from your script, including its version tag.

    If you have multiple scripts to add, you can import the definitions with:

    ska-sdp import scripts my-scripts.yaml
    

    An example file for importing scripts can be found at: Example Script Definitions

  • To run the script, create a processing block, either via the Tango interface, or by creating it directly in the configuration DB with ska-sdp create pb.

3. Finish development and deploy the script

  • Once you are happy with the script, add it to the GitLab CI file (.gitlab-ci.yml) in the root of the repository. You need to add jobs to publish the development and release versions:

    publish-dev-<my-script>:
      extends: .publish-dev
      variables:
        SRCDIR: src/<my-script>
    
    publish-release-<my-script>:
      extends: .publish-release
      variables:
        SRCDIR: src/<my-script>
    

    This will enable the Docker image to be built and pushed to the SKA artefact repository when it is merged into the master branch.

  • Add the script to the script definition file scripts.yaml in the root of the repository. By default the SDP uses this file to populate the script definitions in the configuration DB when it starts up.

  • Create a README.md and add the description and instructions to run your script. Include it in the documentation:

    • create a new file in docs/src/scripts/<my-script>.rst

    • add the following to it:

    .. mdinclude:: ../../../src/<my-script>/README.md
    
    • update docs/src/index.rst

  • Commit the changes to your branch and push to GitLab.

Script Generator

To speed things up for the developer, a Python script has been developed to automatically generate the source files.

The script will generate all the required source files (Python script, requirements, version number, Dockerfile, Makefile, README) for both real-time and batch scripts.

Note: you will need docopt to be installed for the script to run:

pip install docopt

To run the script:

cd templates
python create_script.py <kind> <name>

For example:

python create_script.py realtime my-realtime

Usage

Create source files for real-time or batch script.

Usage:
    create_script.py <kind> <name>
    create_script.py (-h|--help)

Arguments:
    <kind>    Kind of script (realtime or batch)
    <name>    Name of the script to be created