Internals

ska_telmodel.data

class ska_telmodel.data.backend.CARBackend(uri: str, *args, **kwargs)

Represents data in (a mirror of) the SKA central artefact repository. Permissible URI formats:

car:[project name]?[branch]#[directory]
car://[gitlab server]/[project name]?[branch]#[directory]

So for instance:

car:ska-telmodel?master
car://gitlab.com/ska-telescope/ska-telmodel?master#tmdata

The source of truth might still be Gitlab, yet this backend will only work with artefacts that have been uploaded to the CAR. The short form URI will be expanded into the long form automatically.

classmethod backend_name() → str

Returns the name of the backend.

Will be used for the scheme in URIs to identify the backend type of a telescope model data source.

get_uri(pinned: bool) → str

Returns URI for this telescope model data backend

Parameters:

pinned – Attempt to return an URI that will continue to refer to this specific version of telescope model data

Returns:

URI identifying data source

class ska_telmodel.data.backend.DatabaseBackend(uri: str, update: bool = False)

Interface to data stored on an instance of ska-telmodel-db-server.

URI format:

tmdb://[server url]?rev=[timestamp]
classmethod backend_name() → str

Returns the name of this backend, as used in its URI scheme.

exists(key: str) → bool

Check whether a given key exists. :param key: Key to query :returns: True if key exists

get(key: str) → bytes

Get the data stored with the given key. :param key: Key to query :returns: Bytes stored at key

get_uri(pinned: bool) → str

Returns the URI for this telescope model data backend. If pinned is True, returns a URI that will continue to refer to a specific revision of the data, given by either the timestamp specified in the input URI or the current time.

Parameters:

pinned – If True and the input URI did not specify a revision timestamp, the returned URI will be pinned to the current time.

property keys: list[str]

Returns a list of all available keys in the database.

list_keys(key_prefix: str = '') → Iterable[str]

List children keys Yields all keys with prefix “{key_prefix}/” in ascending order. :param key_prefix: Path to query

make_pinned_query(timestamp: datetime) → str

Returns a query string that is pinned to the given timestamp (if the input URI does not already specify a revision). :param timestamp: The revisiontimestamp to pin to. :returns: The pinned query string.

urlunparse(**kwargs) → str

Constructs a URL from the given components, overriding any components specified in kwargs. By default, uses the pinned query evaluated in the constructor (either the input query or the timestamp at construction time).

Parameters:

kwargs – Components to override (scheme, netloc, path, params, query, fragment)

Returns:

The constructed URL as a string.

class ska_telmodel.data.backend.FilesystemBackend(uri: str, update: bool = False)

Retrieves data from a locally accessible file system. URI format:

file://[absolute path]

Note that changes to the file system are outside of our control. Consistency must be ensured externally.

classmethod backend_name() → str

Returns the name of the backend.

Will be used for the scheme in URIs to identify the backend type of a telescope model data source.

copy(key: str, dest: str)

Write key contents to a file.

Raises KeyError if the key does not exist

Parameters:

  • key – Key to query

  • dest – Path of destination file

exists(key: str) → bool

Check whether a given key exists.

Parameters:

key – Key to query

Returns:

True if key exists

get(key: str) → bytes

Get the data stored with the given key

Parameters:

key – Key to query

Returns:

Bytes stored at key

get_uri(pinned: bool) → str

Returns URI for this telescope model data backend

Parameters:

pinned – Attempt to return an URI that will continue to refer to this specific version of telescope model data

Returns:

URI identifying data source

list_keys(key_prefix: str = '') → Iterable[str]

List children keys

Yields all keys with prefix “{key_prefix}/” in ascending order

Parameters:

key_prefix – Path to query

meta(key: str) → dict[str, Any]

Retrieve meta data regarding the given key.

Parameters:

key – Key to query

Returns:

Dictionary with the size, and is_large values

open(key: str, binary: bool = True) → IO[bytes]

Access data at given key as a file-like object

Raises KeyError if the key does not exist

Parameters:

key – Key to query

class ska_telmodel.data.backend.GitlabBackend(uri: str, update: bool = False, gl: gitlab.Gitlab = None, try_nexus: bool = True, nexus_url: str = None, env=None)

Represents data in a GitLab repository. URI format:

gitlab://[gitlab server]/[project name]?[branch]#[directory]

So for instance:

gitlab://gitlab.com/ska-telescope/ska-telmodel?master#tmdata

Would refer to data contained in the ska-telmodel repository itself.

Repositories accessed in this way should make sure to activate the tmdata standard continuous integration stages (see https://gitlab.com/ska-telescope/templates-repository ) to ensure that telescope model data is cached in the SKAO central artefact repository. Once that has been done, this library will never actually query GitLab directly.

Furthermore, this backend will cache all loaded data locally, including resolved Gitlab references (like master in the example above). This especially means that once instantiated, the version of data will be “pinned” even between different instances (and processes). Use the update parameter to ska_telmodel.data.TMData or GitlabBackend respectively to refresh the local cache.

classmethod backend_name() → str

Returns the name of the backend.

Will be used for the scheme in URIs to identify the backend type of a telescope model data source.

copy(key: str, dest: str)

Write key contents to a file.

Raises KeyError if the key does not exist

Parameters:

  • key – Key to query

  • dest – Path of destination file

exists(key: str) → bytes

Check whether a given key exists.

Parameters:

key – Key to query

Returns:

True if key exists

get(key: str) → bytes

Get the data stored with the given key

Parameters:

key – Key to query

Returns:

Data stored at key, or None if it doesn’t exist

get_uri(pinned: bool) → str

Returns URI for this telescope model data backend

Parameters:

pinned – Attempt to return an URI that will continue to refer to this specific version of telescope model data

Returns:

URI identifying data source

list_keys(key_prefix: str = '') → Iterable[str]

List children keys

Yields all keys with prefix “{key_prefix}/” in ascending order. Exception is if the path is empty, in which case all available keys are listed.

Parameters:

key_prefix – Path to query

meta(key: str) → dict[str, Any]

Retrieve meta data regarding the given key.

Parameters:

key – Key to query

Returns:

Dictionary with the size, and is_large values

open(key: str) → IO[bytes]

Access data at given key as a file-like object

Raises KeyError if the key does not exist

Parameters:

key – Key to query

property project_name: str

Get the current project name.

class ska_telmodel.data.backend.MemoryBackend(uri: str, update: bool = False)

Represents in-memory data. URIs should look as follows:

mem://?[key1]=[value1]&[key2]=[value2]

This will directly set the given telescope model data keys to the given values. Useful for testing, and overriding single values in telescope model data.

classmethod backend_name() → str

Returns the name of the backend.

Will be used for the scheme in URIs to identify the backend type of a telescope model data source.

get(key: str) → bytes

Get the data stored with the given key

Parameters:

key – Key to query

Returns:

Bytes stored at key

get_uri(pinned: bool) → str

Returns URI for this telescope model data backend

Parameters:

pinned – Attempt to return an URI that will continue to refer to this specific version of telescope model data

Returns:

URI identifying data source

list_keys(key_prefix: str = '') → Iterable[str]

List children keys

Yields all keys with prefix “{key_prefix}/” in ascending order

Parameters:

key_prefix – Path to query

meta(key: str) → dict[str, Any]

Retrieve meta data regarding the given key.

Parameters:

key – Key to query

Returns:

Dictionary with the size, and is_large values

class ska_telmodel.data.backend.TMDataBackend(uri: str, update: bool = False)

Base class for telescope model data backends

Sub-classes should override backend_name(), then utilise telmodel_backend() to register the telescope model data backend. A minimal implementation should furthermore provide list_keys() and get().

abstract classmethod backend_name() → str

Returns the name of the backend.

Will be used for the scheme in URIs to identify the backend type of a telescope model data source.

copy(key: str, dest: str)

Write key contents to a file.

Raises KeyError if the key does not exist

Parameters:

  • key – Key to query

  • dest – Path of destination file

exists(key: str) → bool

Check whether a given key exists.

Parameters:

key – Key to query

Returns:

True if key exists

abstract get(key: str) → bytes

Get the data stored with the given key

Parameters:

key – Key to query

Returns:

Data stored at key, or None if it doesn’t exist

get_uri(pinned: bool) → str

Returns URI for this telescope model data backend

Parameters:

pinned – Attempt to return an URI that will continue to refer to this specific version of telescope model data

Returns:

URI identifying data source

classmethod is_hidden_file(key: str) → bool

Check whether argument key represents a hidden file

For this to be valid, it needs to: * Have every path segment start with a letter * Have a filename that begins with a dot

Returns:

Validity of key

abstract list_keys(key_prefix: str = '') → Iterable[str]

List children keys

Yields all keys with prefix “{key_prefix}/” in ascending order. Exception is if the path is empty, in which case all available keys are listed.

Parameters:

key_prefix – Path to query

meta(key: str) → dict[str, Any]

Retrieve meta data regarding the given key.

Parameters:

key – Key to query

Returns:

Dictionary with the size, and is_large values

open(key: str) → IO[bytes]

Access data at given key as a file-like object

Raises KeyError if the key does not exist

Parameters:

key – Key to query

classmethod valid_key(key: str) → bool

Check whether this is a valid key we could store data for

For this to be valid, it needs to: * Have every path segment start with a letter * Have no dot in directory names, and a dot in file name

Returns:

Validity of key

classmethod valid_prefix(key: str) → bool

Check whether argument could be a valid prefix to a key

For this to be valid, it needs to: * Have every path segment start with a letter * Have no dot in directory names, and a dot in file name

Returns:

Validity of key

This module concerns itself with adding new data to a current repository.

class ska_telmodel.data.new_data_backend.GitBackend(repo: str = 'ska-telescope/ska-telmodel-data', git_host: str = 'gitlab.com')

This backend uses git as it’s data source, it assumes that the authentication is handled on the host side.

The repo should be one of (in order of preference):

If checkout_location is provided that will be used instead of a temp directory. By default we use ~/.ska-telmodel and place each checkout in there.

add_data(path: Path, key: str | None = None) → None

Copy new file into repo, and run validate on each file.

If path is a directory, the directory structure will also be taken into account when creating the key.

Parameters:

  • path – This can be either a directory of files, or a single file.

  • key – If path is a directory, then key is used as the prefix, if path is a file then this is used as the key.

checkout_branch(name)

Checkout named branch.

commit_transaction() → None

Push changes to branch, if there are no local uncommitted changes.

local_location() → Path

The location of this backend on disk.

start_transaction(name_of_update: str, create_new_branch: bool = True) → None

Create a new clone (if needed), pull the main branch, and create a new branch.

Parameters:

name_of_update – This will become the branch name.

status() → dict[str, list[str]]

Get current status of all new files.

Returns:

the names of the files in different states.

validate(file: Path) → bool

Validates the given file. The file can be anywhere.

Parameters:

file – The path to the file.

Returns:

Whether the file is validated or not.

class ska_telmodel.data.new_data_backend.NewDataBackend

Base class for the data uploading backends.

This class is meant just as a guide to how to use the backends.

abstract add_data(path: Path, key: str | None = None) → None

Add a new data file or directory to the transaction.

Parameters:

  • path – This can be either a directory of files, or a single file.

  • key – If path is a directory, then key is used as the prefix, if path is a file then this is used as the key.

abstract commit_transaction() → None

Save the current transaction and mark it as completed.

abstract local_location() → Path

The location of this backend on disk.

abstract start_transaction(name_of_update: str) → None

Start a new session.

Parameters:

name_of_update – A short description of the update.

abstract status() → dict[str, list[str]]

Get the state of the current session

abstract validate(file: Path) → bool

Validate the given file.