SDP command-line interface

Command Line Interface: ska-sdp

To run the CLI, you must start a shell in the console pod (assuming you have SDP deployed in Kubernetes, to do this follow the instructions in Installing the SDP).

kubectl exec -it ska-sdp-console-0 -n <namespace> -- bash

Once in, to access the help window of ska-sdp, run:

ska-sdp -h

Note that if you need a text editor (e.g. for the edit command), you may need to install one manually.

Command - SDP Object matrix

This is a table/matrix of the existing commands of ska-sdp and what they can do with a specific SDP Object.

Commands:

list
get/watch
create
update/edit
end
delete
import

SDP Objects:

system
pb (processing block)
script (processing script definition)
deployment
eb (execution block)
flow
controller (Tango controller device)
subarray (Tango subarray device)

	system	pb	script	deployment	eb	flow	other
list	Not implemented for system (there is only one)	list all pbs list pbs for a certain date	list all script definitions list a script def of a specific kind (batch or realtime)	list all deployments	list all ebs	list all flows list flows for a certain pb	if `-a` \| `--all`: list all the contents of the Config DB if controller, list the device entry if there is one if subarray, list all subarray device entries Options that work for all objects: if `-v` \| `--values`: list keys with values (or just values) if `--prefix`: list limited to this prefix (for testing purposes)
get/watch	get the value of the system object	get the value of a single key get the values of all pb-related keys for a single pb-id	get the value of a single key	get the value of a single key	get the value of a single key	get the value of a single key get the values of all flows for a single pb and/or flow kind	get and watch also work for other objects. Note: rules for get and watch are the same.
create	n/a	create a pb to run a processing script if `--eb`: add eb parameters for real-time pb	create a key/value pair with prefix of /script	create a deployment of given deployment-id, kind, and parameters	Not implemented for eb	create a flow	Not implemented for Tango devices
update/edit	update/edit the system definition	update/edit the state of a pb with a given pb-id	update/edit a script with a given value	update/edit the state of a deployment with a given dpl-id	update/edit the state of an eb with a given eb-id	update/edit the state of a flow with a given key	update a Tango device entry edit a Tango device entry
delete	n/a	delete all pbs (need confirmation) delete all pb entries for a single pb-id	delete all script defs (need confirmation) delete script def for a single key (kind:name:version)	delete all deployments (need confirmation) delete deployment for a single deployment-id	delete all ebs (need confirmation) delete eb for a single eb-id	delete all flows (need confirmation) delete flow with a single key	if `--prefix`: append prefix in front of path and perform same deletion as listed under SDP object type.
end	n/a	n/a	n/a	n/a	end execution block for a single eb-id (status set to FINISHED)	n/a	if `-c` \| `--cancel`: Cancel the execution block (status set to CANCELLED)
import	import system definition from file, URL, or Telescope Model Data	n/a	import script definitions from file, URL, or Telescope Model Data	n/a	n/a	n/a	n/a

Relevant environment variables

Backend-related:

SDP_CONFIG_BACKEND   Database backend (default etcd3)
SDP_CONFIG_HOST      Database host address (default 127.0.0.1)
SDP_CONFIG_PORT      Database port (default 2379)
SDP_CONFIG_PROTOCOL  Database access protocol (default http)
SDP_CONFIG_CERT      Client certificate
SDP_CONFIG_USERNAME  User name
SDP_CONFIG_PASSWORD  User password

When running ska-sdp edit:

EDITOR    Executable of an existing text editor. Recommended: vi, vim, nano (i.e. command line-based editors)

Usage

> ska-sdp --help

Command line utility for interacting with SKA Science Data Processor (SDP).

Usage:
    ska-sdp COMMAND [options] [SDP_OBJECT] [<args>...]
    ska-sdp COMMAND (-h|--help)
    ska-sdp (-h|--help)

SDP Objects:
    pb           Interact with processing blocks
    script       Interact with available processing script definitions
    deployment   Interact with deployments
    eb           Interact with execution blocks
    flow         Interact with flows
    controller   Interact with Tango controller device
    subarray     Interact with Tango subarray device
    system       Interact with system configuration

Commands:
    list           List information of object from the Configuration DB
    get | watch    Print all the information (i.e. value) of a key in the Config DB
    create         Create a new, raw key-value pair in the Config DB;
                   Run a processing script; Create a deployment
    update         Update a raw key value from CLI
    edit           Edit a raw key value from text editor
    delete         Delete a single key or all keys within a path from the Config DB
    end            Stop/Cancel execution block
    import         Import system or processing script definitions from file or URL

Options:
    --prefix=<prefix>   Global prefix to paths (e.g. /test for testing)

> ska-sdp list --help

List keys (and optionally values) within the Configuration Database.

Usage:
    ska-sdp list (-a |--all) [options]
    ska-sdp list [options] pb [<date>]
    ska-sdp list [options] flow [<pb-id>]
    ska-sdp list [options] script [<kind>]
    ska-sdp list [options] (deployment|eb|controller|subarray)
    ska-sdp list (-h|--help)

Arguments:
    <date>      Date on which the processing block(s) were created. Expected format: YYYYMMDD
                If not provided, all pbs are listed.
    <pb-id>     Processing block id to list flows for. If not provided, all flows are listed
    <kind>      Kind of processing script definition. Batch or realtime.
                If not provided, all scripts are listed.

Options:
    -h, --help         Show this screen
    -q, --quiet        Cut back on unnecessary output
    -a, --all          List the contents of the Config DB, regardless of object type
    -v, --values       List all the values belonging to a key in the config db
    --prefix=<prefix>  Path prefix (in addition to standard Config paths, e.g. /test for testing)

> ska-sdp (get|watch) --help

Get/Watch all information of a single key in the Configuration Database.

Usage:
    ska-sdp (get|watch) [options] system
    ska-sdp (get|watch) [options] <key>
    ska-sdp (get|watch) [options] pb <pb-id>
    ska-sdp (get|watch) [options] eb <eb-id>
    ska-sdp (get|watch) [options] deployment <dpl-id>
    ska-sdp (get|watch) [options] flow <flow-id>
    ska-sdp (get|watch) (-h|--help)

Arguments:
    <key>       Key within the Config DB.
                To get the list of all keys:
                    ska-sdp list -a
    <pb-id>     Processing block id to list all entries and their values for.
                Else, use key to get the value of a specific pb.
    <eb-id>     Execution block id to list all entries and their values for.
                Else, use key to get the value of a specific eb.
    <dpl-id>    Deployment id to list all entries and their values for.
                Else, use key to get the value of a specific deployment.
    <flow-id>   Flow to list all entries and their values for. Expected format:
                pb-id:kind:name
                Else, use key to get the value of a specific pb.

Options:
    -h, --help    Show this screen
    -q, --quiet   Cut back on unnecessary output

> ska-sdp create --help

Create SDP objects (deployment, script) in the Configuration Database.
Create a processing block to run a script.

Usage:
    ska-sdp create [options] pb <script> [<parameters>] [--eb=<eb-parameters>]
    ska-sdp create [options] deployment <item-id> <kind> <parameters>
    ska-sdp create [options] script <item-id> <value>
    ska-sdp create [options] flow <parameters>
    ska-sdp create (-h|--help)

Arguments:
    <script>            Script that the processing block will run, in the format:
                            kind:name:version
    <parameters>        Optional parameters for a script, with expected format:
                            '{"key1": "value1", "key2": "value2"}'
                        For deployments, expected format:
                            '{"chart": <chart-name>, "values": <dict-of-values>}'
                        For flow, expected format:
                            '{"key":{}, "sink":{}, "sources":[], "data_model":""}'
    <eb-parameters>     Optional eb parameters for a real-time script
    <item-id>           Id of the new deployment or script
    <kind>              Kind of the new deployment ("helm" or "slurm")
    <value>             Script definition

Options:
    -h, --help     Show this screen
    -q, --quiet    Cut back on unnecessary output

Example:
    ska-sdp create pb realtime:vis-receive:0.0.0 '{"": ""}'

Note: You cannot create processing blocks apart from when they are called to run a script.
Execution blocks are not allowed to be created on their own, but only
when you create a realtime processing block with --eb option on.

> ska-sdp (update|edit) --help

Update the value of a single key or pb/eb/deployment state.
Can either update from CLI, or edit via a text editor.

Usage:
    ska-sdp update [options] (eb-state|dpl-state|pb-state|flow-state) <item-id> <value>
    ska-sdp update [options] script <item-id> <value>
    ska-sdp update [options] system <value>
    ska-sdp update [options] controller <value>
    ska-sdp update [options] subarray <item-id> <value>
    ska-sdp edit (eb-state|dpl-state|pb-state|flow-state) <item-id>
    ska-sdp edit script <item-id>
    ska-sdp edit system
    ska-sdp edit controller
    ska-sdp edit subarray <item-id>
    ska-sdp (update|edit) (-h|--help)

Arguments:
    <item-id>   id of the script, eb, deployment, processing block or subarray
    <value>     Value to use for update, with format '{"key":"value"}'

Options:
    -h, --help    Show this screen
    -q, --quiet   Cut back on unnecessary output

Note:
    ska-sdp edit needs an environment variable defined:
        EDITOR: Has to match the executable of an existing text editor
                Recommended: vi, vim, nano (i.e. command line-based editors)
        Example: EDITOR=vi ska-sdp edit <key>
    Processing blocks, execution blocks and deployments cannot be changed,
    apart from their state.

Example:
    ska-sdp edit eb-state eb-test-20210524-00000
        --> key that's edited: /eb/eb-test-20210524-00000/state
    ska-sdp edit script batch:test:0.0.0
        --> key that's edited: /script/batch:test:0.0.0
    ska-sdp edit pb-state some-pb-id-0000
        --> key that's edited: /pb/some-pb-id-0000/state

> ska-sdp delete --help

Delete a key from the Configuration Database.

Usage:
    ska-sdp delete (-a|--all) [options] (pb|script|eb|deployment|flow|prefix)
    ska-sdp delete [options] (pb|eb|deployment|flow) <item-id>
    ska-sdp delete [options] script <script>
    ska-sdp delete (-h|--help)

Arguments:
    <item-id>   ID of the processing block, deployment, execution block, or flow
    <script>    Script definition to be deleted. Expected format: kind:name:version
    prefix      Use this "SDP Object" when deleting with a non-object-specific, user-defined prefix

Options:
    -h, --help             Show this screen
    -q, --quiet            Cut back on unnecessary output
    --prefix=<prefix>      Path prefix (if other than standard Config paths, e.g. for testing)

> ska-sdp end --help

End execution block in the configuration database.
By default it sets the status to FINISHED. If the --cancel flag is set, it sets
the status to CANCELLED.

Usage:
    ska-sdp end eb <eb-id> [options]
    ska-sdp end (-h|--help)

Arguments:
<eb-id>    ID of execution block to end

Options:
    -c, --cancel  Cancel the execution block
    -h, --help    Show this screen
    -q, --quiet   Cut back on unnecessary output

> ska-sdp import --help

Import processing script definitions into the Configuration Database.
It currently supports input as a yaml file or URL containing a json string.
Note that the input can contain a system defintion, a single script,
multiple versions of a script, or multiple scripts.

Usage:
    ska-sdp import (system|script|scripts) [options] <file-or-url>
    ska-sdp import (-h|--help)

Arguments:
    <file-or-url>   File or URL to import script definitions from.
                    URL can also be a pointer to Telescope Model Data in the
                    format of: "tmdata::<tmdata-source-url>::<tmdata-file>"
                    <tmdata-file> is optional and defaults to "ska-sdp/scripts/scripts.yaml"

Options:
    -h, --help      Show this screen
    --sync          Delete scripts not in the input (only works for script/scripts)

Example flow definition

When creating a new Flow entity using the create command, the parameters that describe the flow must be given as a dictionary. There are various kinds of Flow sink, which can be checked in the Flow entry on the API Entities page. The parameters are checked against the schema and mistakes will show up as Pydantic errors. Note that in order to create the correct Flow kind, the kind parameter must be supplied in both the key and the sink.

A simple example for a data-product flow is:

ska-sdp create flow '{"key": {"pb_id":"pb-test-20250119-00000","kind":"data-product","name":"pointing-offset"}, "sink": {"kind":data-product,"data_dir":"/wow","paths":[]}, "sources":[], "data_model":"PointingTable"}'

Example system definitions file

You can use a Yaml file to import system definitions into the Config DB. An example system definitions file looks like

version: "0.17.0"
components:
    lmc-controller:
        devicename: "test-sdp/controller/0"
        image: "artefact.skao.int/ska-sdp-lmc"
        version: "0.24.0"
    lmc-subarray-01:
        devicename: "test-sdp/subarray/01"
        image: "artefact.skao.int/ska-sdp-lmc"
        version: "0.24.0"
dependencies:
    ska-tango-base:
        repository: "https://artefact.skao.int/helm-internal"
        version: "0.4.6"

After the import, you can check via

ska-sdp get system

It will output a list of SDP components that have been imported.

Example script definitions file

You can also use a script definitions file to import processing scripts into the Config DB. An example script definitions file looks like

scripts:
- kind: realtime
  name: test_realtime
  version: 0.7.0
  image: artefact.skao.int/ska-sdp-script-test-realtime:0.7.0
  sdp_version: ">=0.21.0"
- kind: batch
  name: test_batch
  version: 0.5.0
  image: artefact.skao.int/ska-sdp-script-test-batch:0.5.0
  sdp_version: ">=0.19.0, <0.22.0"

Both YAML and JSON files are accepted. After the import, you can check via

ska-sdp list script

It will output a list of processing scripts that are available to use.