Usage Guide

Data

Telescope model data is stored as objects identified as keys. Each key takes the following form:

[domain]/([sub-domain]/)*[name].[type]

Where

  • [domain]/ specifies the coarse top-level telescope model data domain.

  • [sub-domain]/ provides further hierarchical data sub-categories

  • [name] associates a name with the telescope model data.

  • [type] identifies the file type, which is used to identify the kind of file contents. The library currently supports json and yaml.

Names should be chosen for being self-describing and stable long-term. Top-level domains:

environment/...   # Environment telescopes are deployed in
instrument/...    # Telescopes and their equipment
software/...      # Software deployed to the telescopes

To retrieve a particular piece of data from the telescope model, csimply construct an ska_telmodel.data.TMData object and use the [] operator to access:

from ska_telmodel.data import TMData
tmdata = TMData()
print(tmdata['instrument/ska1_low/layout/low-layout.json'].get_dict())

This works because the telescope model comes with a number of default sources that will be checked for matching telescope model data. ska_telmodel.data.TMObject.get_dict() automatically parses and converts JSON and YAML documents, but you can also get the raw data using ska_telmodel.data.TMObject.get(), or open or copy the contents as a file using ska_telmodel.data.TMObject.open() or ska_telmodel.data.TMObject.copy() respectively.

To get an idea what is contained in a particumlar part of the telescope model data tree, simply iterate over it (equivalent to ska_telmodel.cli.cmd_ls()):

from ska_telmodel.data import TMData
tmdata = TMData()
for key in tmdata['instrument']:
    print(key)

Logically, the [] operator constructs a sub-set of all telescope model data. If the key is a valid object name (i.e. has an extension, so contains a '.') this subset is assumed to contain only a single object, and the [] operator will return a ska_telmodel.data.TMObject instance.

Data Sources

Telescope model data can be retrieved from a list of sources, which can be specified to the ska_telmodel.data.TMData constructor, using the SKA_TELMODEL_SOURCES environment variable or left to in-built ska_telmodel.data.sources.DEFAULT_SOURCES. Each source is represented as an URI that specifies the source of truth for some portion of telescope model data.

The following telescope model data backends are currently supported:

The simplest example would be to utilise ska_telmodel.data.backend.MemoryBackend to set a key directly:

from ska_telmodel.data import TMData
tmdata = TMData(['mem://?test.txt=test_data'])
print(tmdata['test.txt']) # -> b"test_data\n"

This can also be configured using environment variables:

import os
from ska_telmodel.data import TMData
os.environ['SKA_TELMODEL_SOURCES'] = 'mem://?test.txt=test_data'
tmdata = TMData()
print(tmdata['test.txt']) # -> b"test_data\n"

You would typically do this from outside your program, see the documentation for ska_telmodel.cli.cmd_pin() and ska_telmodel.cli.cmd_cp() for examples.

A more complex example would be to retrieve data from Gitlab using ska_telmodel.data.backend.GitlabBackend:

from ska_telmodel.data import TMData
gl_uri = 'car://gitlab.com/ska-telescope/ska-telmodel?master#tmdata'
tmdata = TMData([gl_uri])
print(tmdata['software/tango/dsh/DishManager.yaml'])

This will retrieve data directly from the telescope model library repository.

Note that external telescope model data sources using ska_telmodel.data.backend.GitlabBackend or ska_telmodel.data.backend.CARBackend will cache data locally in order to prevent repeated requests to servers. This means that if we reference a Gitlab branch (like master in the example), the telescope model data in the cache might go out of sync with the server.

This is intentional, as it means that we provide a consistent view of telescope model data as long as possible. It is generally best to use “pinned” sources (see ska_telmodel.cli.cmd_pin()), but in day-to-day usage, you can simply use the -U flag as documented in Command Line Usage or (less preferably) the update option to ska_telmodel.data.TMData to occassionally refresh the cache as needed. The library will occassionally check for and warn about stale caches.

Adding a New Gitlab Data Source

If you want others to be able to view data in your GitLab repository using ska_telmodel.data.backend.GitlabBackend or ska_telmodel.data.backend.CARBackend, first you will need to place the data you wish to export in a top level dictionary in your repo named tmdata. For example:

/tmdata/instrument/mccs_configuration/config_file_low.json
/tmdata/instrument/mccs_configuration/config_file_mid.json

Important to note:

  • Try to use a directory structure that is compatible with domains (see above) and is reasonably likely to remain stable.

  • Currently only .json and .yaml files are accepted, and you should have schemas associated with them.

Next add telescope model data support to your top-level Makefile as documented in https://developer.skao.int/projects/ska-cicd-makefile/en/latest/README.html :

include .make/tmdata.mk

At this point you should be able to verify that make tmdata-package will result in both a tmtree.json and a tmdata.tar.gz file getting created in build/tmdata. Next add the packaging and publishing stage to your Gitlab pipeline by adding the following lines to the .gitlab-ci.yml file as documented in https://developer.skao.int/projects/templates-repository/en/latest/README.html :

- project: 'ska-telescope/templates-repository'
  file: 'gitlab-ci/includes/tmdata.gitlab-ci.yml'

Now once you merge these changes into the main branch, others will be able to access this data by specifying your repository as the source:

$ ska-telmodel --sources=car:mccs/ska-low-mccs?main ls
instrument/mccs_configuration/config_file_low.json
instrument/mccs_configuration/config_file_mid.json

Branches other than main will also work, just adjust the URL accordingly. However by default the GitLab pipeline will only upload the TMData package to the artefact repository on the main branch as well as tags.

If you want the data to be accessible without passing command line parameters, make a merge request to the ska-telmodel repository ( https://gitlab.com/ska-telescope/ska-telmodel ) that adds your repository address the src/ska_telmodel/data/source.py file. This makes your telescope model data available “globally”:

$ ska_telmodel ls
[...]
instrument/mccs_configuration/config_file_low.json
instrument/mccs_configuration/config_file_mid.json

Schemas

Schemas check JSON-like objects for conformance, e.g. nested dictionaries containing primitives and lists. They especially have a JSON schema representation - though ska_telmodel.schema.validate() will generally implement more thorough checks.

All schemas are identified by an URI of the form:

https://schema.skao.int/ska-[subsystem]-[interface]/[major].[minor]

The entire URI should be lower-case alphanumerical. The [subsystem] identifies the leading party for maintaining the schema, and [interface] the concrete interface implemented. Depending on context, this might either be data produced or consumed by the sub-system in question.

Versioning should follow semantic versions: Changes in minor version indicate backwards-compatible changes such as adding new fields or otherwise introducing additional accepted schemas. Changes that break backwards compatibility should change the major version.

You can use the URIs with ska_telmodel.schema.validate() to validate data:

from ska_telmodel.data import TMData
from ska_telmodel.schema import validate

uri = "https://schema.skao.int/ska-telmodel-layout-location/0.0"
layout_dict = TMData()['instrument/ska1_low/layout/low-layout.json'].get_dict()
validate(uri, layout_dict)

Furthermore you can use ska_telmodel.schema.example_by_uri() to retrieve examples of certain schemas (which are replicated in the schema section of this documentation).