SDP command-line interface

Command Line Interface: ska-sdp

To run the CLI, you must start a shell in the console pod (assuming you have SDP deployed in Kubernetes/Minikube, for instructions follow: SDP standalone).

kubectl exec -it ska-sdp-console-0 -n <namespace> -- bash

Once in, to access the help window of ska-sdp, run:

ska-sdp -h

Command - SDP Object matrix

This is a table/matrix of the existing commands of ska-sdp and what they can do with a specific SDP Object.

Commands:

  • list

  • get/watch

  • create

  • update/edit

  • end

  • delete

  • import

SDP Objects:

  • pb (processing block)

  • script (processing script definition)

  • deployment

  • eb (execution block)

  • controller (Tango controller device)

  • subarray (Tango subarray device)

pb

script

deployment

eb

other

list

  • list all pbs

  • list pbs for a certain date

  • list all script definitions

  • list a script def of a specific kind (batch or realtime)

list all deployments

list all ebs

  • if -a | –all: list all the contents of the Config DB

  • if -v | –values: list keys with values (or just values)

  • if –prefix: list limited to this prefix (for testing purposes)

  • if controller, list the device entry if there is one

  • if subarray, list all subarray device entries

get/watch

  • get the value of a single key

  • get the values of all pb-related keys for a single pb-id

get the value of a single key

get the value of a single key

get the value of a single key

Note: rules for get and watch are the same

create

  • create a pb to run a processing script

  • if –eb: add eb parameters for real-time pb

create a key/value pair with prefix of /script

create a deployment of given deployment-id, kind, and parameters

create a key/value pair with prefix of /eb

Not implemented for Tango devices

update/edit

update/edit the state of a pb with a given pb-id

  • update a given key with a given value

  • edit a given key

  • update a given key with a given value

  • edit a given key

  • update a given key with a given value

  • edit a given key

  • update a Tango device entry

  • edit a Tango device entry

delete

  • delete all pbs (need confirmation)

  • delete all pb entries for a single pb-id

  • delete all script defs (need confirmation)

  • delete script def for a single key (kind:name:version)

  • delete all deployments (need confirmation)

  • delete deployment for a single deployment-id

  • delete all ebs (need confirmation)

  • delete eb for a single eb-id

  • if –prefix: append prefix in front of path and perform same

  • deletion as listed under SDP object type.

end

n/a

n/a

n/a

end execution block for a single eb-id (status set to FINISHED)

if -c | –cancel: Cancel the execution block (status set to CANCELLED)

import

n/a

import script definitions from file or URL

n/a

n/a

Relevant environment variables

Backend-related:

SDP_CONFIG_BACKEND   Database backend (default etcd3)
SDP_CONFIG_HOST      Database host address (default 127.0.0.1)
SDP_CONFIG_PORT      Database port (default 2379)
SDP_CONFIG_PROTOCOL  Database access protocol (default http)
SDP_CONFIG_CERT      Client certificate
SDP_CONFIG_USERNAME  User name
SDP_CONFIG_PASSWORD  User password

When running ska-sdp edit:

EDITOR    Executable of an existing text editor. Recommended: vi, vim, nano (i.e. command line-based editors)

Usage

> ska-sdp --help

Command line utility for interacting with SKA Science Data Processor (SDP).

Usage:
    ska-sdp COMMAND [options] [SDP_OBJECT] [<args>...]
    ska-sdp COMMAND (-h|--help)
    ska-sdp (-h|--help)

SDP Objects:
    pb           Interact with processing blocks
    script       Interact with available processing script definitions
    deployment   Interact with deployments
    eb           Interact with execution blocks
    controller   Interact with Tango controller device
    subarray     Interact with Tango subarray device

Commands:
    list           List information of object from the Configuration DB
    get | watch    Print all the information (i.e. value) of a key in the Config DB
    create         Create a new, raw key-value pair in the Config DB;
                   Run a processing script; Create a deployment
    update         Update a raw key value from CLI
    edit           Edit a raw key value from text editor
    delete         Delete a single key or all keys within a path from the Config DB
    end            Stop/Cancel execution block
    import         Import processing script definitions from file or URL

> ska-sdp list --help

List keys (and optionally values) within the Configuration Database.

Usage:
    ska-sdp list (-a |--all) [options]
    ska-sdp list [options] pb [<date>]
    ska-sdp list [options] script [<kind>]
    ska-sdp list [options] (deployment|eb|controller|subarray)
    ska-sdp list (-h|--help)

Arguments:
    <date>      Date on which the processing block(s) were created. Expected format: YYYYMMDD
                If not provided, all pbs are listed.
    <kind>      Kind of processing script definition. Batch or realtime.
                If not provided, all scripts are listed.

Options:
    -h, --help         Show this screen
    -q, --quiet        Cut back on unnecessary output
    -a, --all          List the contents of the Config DB, regardless of object type
    -v, --values       List all the values belonging to a key in the config db; default: False
    --prefix=<prefix>  Path prefix (if other than standard Config paths, e.g. for testing)

> ska-sdp (get|watch) --help

Get/Watch all information of a single key in the Configuration Database.

Usage:
    ska-sdp (get|watch) [options] <key>
    ska-sdp (get|watch) [options] pb <pb-id>
    ska-sdp (get|watch) (-h|--help)

Arguments:
    <key>       Key within the Config DB.
                To get the list of all keys:
                    ska-sdp list -a
    <pb-id>     Processing block id to list all entries and their values for.
                Else, use key to get the value of a specific pb.

Options:
    -h, --help    Show this screen
    -q, --quiet   Cut back on unnecessary output

> ska-sdp create --help

Create SDP objects (deployment, script, eb) in the Configuration Database.
Create a processing block to run a script.

Usage:
    ska-sdp create [options] pb <script> [<parameters>] [--eb=<eb-parameters>]
    ska-sdp create [options] deployment <item-id> <kind> <parameters>
    ska-sdp create [options] (script|eb) <item-id> <value>
    ska-sdp create (-h|--help)

Arguments:
    <script>            Script that the processing block will run, in the format:
                            kind:name:version
    <parameters>        Optional parameters for a script, with expected format:
                            '{"key1": "value1", "key2": "value2"}'
                        For deployments, expected format:
                            '{"chart": <chart-name>, "values": <dict-of-values>}'
    <eb-parameters>     Optional eb parameters for a real-time script
    <item-id>           Id of the new deployment, script or eb
    <kind>              Kind of the new deployment (currently "helm" only)

Options:
    -h, --help     Show this screen
    -q, --quiet    Cut back on unnecessary output

Example:
    ska-sdp create eb eb-test-20210524-00000 '{"test": true}'
    Result in the config db:
        key: /eb/eb-test-20210524-00000
        value: {"test": true}

Note: You cannot create processing blocks apart from when they are called to run a script.

> ska-sdp (update|edit) --help

Update the value of a single key or processing block state.
Can either update from CLI, or edit via a text editor.

Usage:
    ska-sdp update [options] (script|eb|deployment) <item-id> <value>
    ska-sdp update [options] pb-state <item-id> <value>
    ska-sdp update [options] controller <value>
    ska-sdp update [options] subarray <item-id> <value>
    ska-sdp edit (script|eb|deployment) <item-id>
    ska-sdp edit pb-state <item-id>
    ska-sdp edit controller
    ska-sdp edit subarray <item-id>
    ska-sdp (update|edit) (-h|--help)

Arguments:
    <item-id>   id of the script, eb, deployment, processing block or subarray
    <value>     Value to update the key/pb state with.

Options:
    -h, --help    Show this screen
    -q, --quiet   Cut back on unnecessary output

Note:
    ska-sdp edit needs an environment variable defined:
        EDITOR: Has to match the executable of an existing text editor
                Recommended: vi, vim, nano (i.e. command line-based editors)
        Example: EDITOR=vi ska-sdp edit <key>
    Processing blocks cannot be changed, apart from their state.

Example:
    ska-sdp edit eb eb-test-20210524-00000
        --> key that's edited: /eb/eb-test-20210524-00000
    ska-sdp edit script batch:test:0.0.0
        --> key that's edited: /script/batch:test:0.0.0
    ska-sdp edit pb-state some-pb-id-0000
        --> key that's edited: /pb/some-pb-id-0000/state

> ska-sdp delete --help

Delete a key from the Configuration Database.

Usage:
    ska-sdp delete (-a|--all) [options] (pb|script|eb|deployment|prefix)
    ska-sdp delete [options] (pb|eb|deployment) <item-id>
    ska-sdp delete [options] script <script>
    ska-sdp delete (-h|--help)

Arguments:
    <item-id>   ID of the processing block, or deployment, or execution block
    <script>    Script definition to be deleted. Expected format: kind:name:version
    prefix      Use this "SDP Object" when deleting with a non-object-specific, user-defined prefix

Options:
    -h, --help             Show this screen
    -q, --quiet            Cut back on unnecessary output
    --prefix=<prefix>      Path prefix (if other than standard Config paths, e.g. for testing)

> ska-sdp end --help

End execution block in the configuration database.
By default it sets the status to FINISHED. If the --cancel flag is set, it sets
the status to CANCELLED.

Usage:
    ska-sdp end eb <eb-id> [options]
    ska-sdp end (-h|--help)

Arguments:
<eb-id>    ID of execution block to end

Options:
    -c, --cancel  Cancel the execution block
    -h, --help    Show this screen
    -q, --quiet   Cut back on unnecessary output

> ska-sdp import --help

Import processing script definitions into the Configuration Database.

Usage:
    ska-sdp import scripts [options] <file-or-url>
    ska-sdp import (-h|--help)

Arguments:
    <file-or-url>      File or URL to import script definitions from.

Options:
    -h, --help          Show this screen
    --sync              Delete scripts not in the input

Example script definitions file

You can also use a script definitions file to import processing scripts into the Config DB. An example script definitions file looks like

scripts:
- kind: realtime
  name: test_realtime
  version: 0.2.2
  image: artefact.skao.int/ska-sdp-script-test-realtime:0.2.2
- kind: batch
  name: test_batch
  version: 0.2.2
  image: artefact.skao.int/ska-sdp-script-test-batch:0.2.2

Both YAML and JSON files are accepted. After the import, you can check via

ska-sdp list script

It will output a list of processing scripts that are available to use.