Getting Started

Installation

Install using pip from the SKAO central artefact repository:

$ pip install --extra-index-url https://artefact.skao.int/repository/pypi-internal/simple ska-telmodel

Simple usage

List data

You can now use the command line utility to list default telescope model data:

$ ska-telmodel ls
instrument/mccs-configuration/station_export_w2.json
instrument/mccs-configuration/antenna_export_w2.json
instrument/ska1_low/layout/low-layout.json
instrument/ska1_low/layout/data.json
instrument/ska1_low/layout/README.md
[...]

You can achieve the same thing from Python as follows:

from ska_telmodel.data import TMData
for key in TMData(): print(key)

Retrieve data

You can easily retrieve data from the command line as well:

$ ska-telmodel cat instrument/ska1_low/layout/README.md

SKA Low layout
--------------
[...]

Again, the same can be achieved from Python:

from ska_telmodel.data import TMData
print(TMData()['instrument/ska1_low/layout/README.md'].get().decode())

For JSON or YAML data, you can especially retrieve it parsed:

print(TMData()['software/tango/ska_wide/Guidelines.yaml'].get_dict())
# -> [{'class': None, [...]

Data sources

Local directory

ska-telmodel has a number of default data sources built-in, which we have been querying above. However, you can override this. For instance, you can use a local directory as a source:

$ mkdir tmdata_demo
$ echo Test! > tmdata_demo/test.txt
$ ska-telmodel ls --sources=file://tmdata_demo
test.txt
$ ska-telmodel cat --sources=file://tmdata_demo test.txt
Test!

This works similarly from Python:

from ska_telmodel.data import TMData
tmdata = TMData(['file://tmdata_demo'])
print(tmdata['test.txt'].get().decode())
# -> Test!

A useful pattern is to use this to create a local copy of telescope model data (see ska_telmodel.cli.cmd_cp()).

Gitlab & CAR sources

You can also use any Gitlab directory as a source:

$ ska-telmodel ls --sources=gitlab://gitlab.com/ska-telescope/ska-telmodel?master#tmdata/software
UserWarning: gitlab://gitlab.com/ska-telescope/ska-telmodel?master#tmdata/software not cached in SKA CAR - make sure to add tmdata CI!
  warnings.warn(warning)

tango/dsh/DishManager.yaml
tango/ska_wide/Guidelines.yaml
tango/ska_wide/SKABaseDevice.yaml
tango/ska_wide/SKAMaster.yaml

This is useful for development, but as the warning indicates should not be used seriously, as Gitlab will eventually start blocking these kinds of requests. A better approach is to use the SKAO central artefact repository (CAR) as the source:

$ ska-telmodel ls --sources=car:ska-telmodel?master
instrument/ska1_mid/validation/mid-validation-constants.json
software/tango/ska_wide/Guidelines.yaml
software/tango/ska_wide/SKAMaster.yaml
software/tango/ska_wide/SKABaseDevice.yaml
software/tango/dsh/DishManager.yaml

Using the car: URI we are now referring to an archive artefact, typically mirroring the contents of a repository (see Adding a New Gitlab Data Source).

Dynamic sources

The source URIs given above point to dynamic branches (master), which means that the results of queries against telescope model data might change. For instance we can go:

$ echo Test! > tmdata_demo/test.txt
$ git switch -c my_test_branch
$ git add tmdata_demo/test.txt; git commit -m "Telescope model data test"; git push my_test_branch origin
$ export SKA_TELMODEL_SOURCES=gitlab://gitlab.com/ska-telescope/ska-telmodel?my_test_branch#tmdata_demo
$ ska-telmodel ls
test.txt
$ ska-telmodel cat test.txt
Test!
$ echo Test2! > tmdata_demo/test.txt
$ git add tmdata_demo/test.txt; git commit -m "Telescope model data test 2"; git push my_test_branch origin
$ ska-telmodel cat test.txt
Test!
$ ska-telmodel cat --update test.txt
Test2!

Note that the result of our query changed - albeit only after we passed --update, which forced a refresh of the cache. A CAR data source would have the same behaviour if a new package was uploaded by a CI pipeline.

In Python we would achieve the equivalent as follows:

from ska_telmodel.data import TMData
sources = ['gitlab://gitlab.com/ska-telescope/ska-telmodel?my_test_branch#tmdata_demo']
tmdata = TMData(sources, update=True)
print(['test.txt'].get().decode())

Pinning dynamic sources

This dynamic behaviour might be useful in development, but when running code in testing or production, we would like more reproduceability. This is why it is a good idea to “pin” dynamic sources to a specific version. One approach is to refer to a fixed “tag”:

$ ska-telmodel --sources=car:ska-telmodel?1.5.0 ls
software/tango/ska_wide/Guidelines.yaml
software/tango/ska_wide/SKAMaster.yaml
software/tango/ska_wide/SKABaseDevice.yaml
software/tango/dsh/DishManager.yaml

Now we are effectively referring to a “telescope model data release”, which is permanently stored in the CAR and will always give the same result. Note that every repository publishing telescope model data might have its own independent version history, and there’s especially no connection to the version of the telescope model data library.

Another approach is to “pin” sources, which resolves them to hashes:

$ export `ska-telmodel -U pin`
Using car:ska-telmodel-data?~9d576afb2f8980bab1fea5d82fa80ddfa91fba21
Using car:ska-telmodel?~719f0146df1de15dfaaa1780847de656ce35c29a
Using car:mccs/ska-low-mccs?~6d98ac66b188d9943b2af19e3e5f2f317da384e8
$ echo $SKA_TELMODEL_SOURCES
car:ska-telmodel-data?~9d576afb2f8980bab1fea5d82fa80ddfa91fba21,car:ska-telmodel?~719f0146df1de15dfaaa1780847de656ce35c29a,car:mccs/ska-low-mccs?~6d98ac66b188d9943b2af19e3e5f2f317da384e8

In Python we would achieve the same as follows:

from ska_telmodel.data import TMData
sources = TMData(update=True).get_sources(pinned=True)
print(sources)
# -> ['car:ska-telmodel-data?~9d576afb2f8980bab1fea5d82fa80ddfa91fba21', 'car:ska-telmodel?~719f0146df1de15dfaaa1780847de656ce35c29a', 'car:mccs/ska-low-mccs?~6d98ac66b188d9943b2af19e3e5f2f317da384e8']

At this point we would be able to pass sources to a different component (e.g. a configured sub-system):

# Set telescope model data to use, issue call to other component
config['sources'] = tmdata.get_sources(pinned=True)
config['layout_key'] = 'instrument/ska1_low/layout/data.json'
otherComponent.Command(json.dumps(config))

Now another component (e.g. Tango device) could get the data pointed at as follows:

def Commnand(self, config_str):
    config = json.loads(config_str)
    tmdata = TMData(config['sources'])
    layout = tmdata[config['layout_key']]

At this point we could be sure that the second piece of code has exactly the same view of telescope model data - regardless of any updates to telescope model data that might have happened in the meantime.

Permanently adding or changing files

In Dynamic sources we used a Gitlab source to quickly add a file, but this is not how you would add files to telescope model data permanently. As explained in the last section, to add data long-term we want to make them part of telescope model data “releases” persisted in the central artefact repository (such as car:ska-telmodel?1.5.0).

The idea is that any SKAO repository can release such telescope model data packages, similar to how any repository can publish (say) Python packages. For instance, the following repositories currently publish telescope model data:

You can view the information coming from these repositories as usual:

$ ska-telmodel --sources=car:mccs/ska-low-mccs?master ls
instrument/mccs-configuration/station_export_w2.json
instrument/mccs-configuration/antenna_export_w2.json

To add your own information, you need to:

  1. Identify the repository to add the information to. If your telescope model data does not fit into an existing repostory that publishes telescope model data, check Adding a New Gitlab Data Source for how to set up a new repository to publish telescope model data.

  2. Add the data to the tmdata folder in the repository, e.g. using a merge request. Make sure you choose a good path within it, because it will be global, see Data. Once merged, you should be able to see your file using ska-telmodel --sources=car:ska-your-repo?main (assuming your main branch is called main, otherwise master)

  3. Optional: Release your repository (i.e. create a tag) to create a versioned telescope model data package, which can then be accessed using ska-telmodel --sources=car:ska-your-repo?a.b.c where a.b.c is the release version.

Further information

For more in-depth guides, check Usage Guide. There is also an SKAO Slack channel for helping users and developers of the SKA telescope model - #help-telmodel.