Entities and Operations

Note

All code examples below work on a txn Transaction object created like this:

for txn in Config(backend="memory").txn():
    # example code

Overview

The SDP Configuration Database manages entities of different types. Each type of entity is stored under a well-determined prefix or path. Individual entities within a certain type are identified by a key, which is composed by one or more key parts. Each key part accepts values that are constrained by a pattern. Additionally, entities of certain types have an associated owner and state. An entity is said to be alive if its owner entry exists in the database.

The Transaction class provides a series of attributes (e.g., Transaction.deployment) through which users are meant to operate on the different entities that are stored in the database. This section describes how this interaction takes place.

Entity models

Note

Support for entity model classes has been added, but not all entities have been modelled. Check the Supported entities section for information of which entity types are currently modelled. Long-term the aim is to model all of them.

Historically the SDP Configuration Database has dealt with plain dict objects as entity values. On top of that, two classes modelling ProcessingBlock and Deployment had been provided to users allowing for conversion from/to dictionaries for operating with the (old, dict-based) Transaction API.

Entities now are expected to be modelled as individual classes using pydantic. This has several benefits:

Provides users with a well-defined in-memory representation for each entity type.

Makes it possible to easily distinguish between entities of different types.

Allows to model restrictions on each field, triggering validation errors when invalid content is provided.

When an entity is modelled using a pydantic model class, the corresponding operations described below are defined in terms of instances of that class rather than plain dictionaries. The model class also defines the entity key as a field that should be modelled itself with pydantic.

Operating on entities

To access entities of a particular type users need to operate on the corresponding attribute of Transaction. See Supported entities for a list of all supported entities and their corresponding attributes. For example, to operate on processing block entities one selects the processing_block attribute:

>>> txn.processing_block  
<ska_sdp_config.operations.processing_block.ProcessingBlockOperations object at ...>

These attributes provide operations on entities of that type. There are two types of operations: those that act over individual entities, and those that operate over the whole set of entities for a given entity type.

The `self` attribute

The Transaction class has a special self attribute pointing to the operations for the entity pointed to by the owned_entity argument of the Config constructor. The argument is a two-tuple with:

The entity type as a Transaction attribute name

The full key of the entity in question.

See Supported entities for references for both values.

For example, the following Config class is configured to get convenient access to the test component:

cfg2 = Config(backend="memory", owned_entity=("component", "test"))
for txn in cfg2.txn():
   print(txn.self.path)

/component/test

If no identity is provided (like in the case of the Config object used in these examples), self will be None.

>>> txn.self is None
True

`dict` entities

Indexing

To operate on an individual entity modelled by a plain dictionary users need to “index” into the entity type to identify the entity of interest.

This indexing is done by invoking the corresponding attribute with the key information that identifies an individual entity. For example, to operate over the Controller device component identified by "lmc-controller" one calls the component attribute with the key:

>>> txn.component(key="lmc-controller")
<EntityOperations path="/component/lmc-controller">
>>> txn.component(component_name="lmc-controller")
<EntityOperations path="/component/lmc-controller">
>>> txn.component("lmc-controller")
<EntityOperations path="/component/lmc-controller">

Note how here we can provide either:

The special key keyword argument. This specifies the full key as a single value (see Overview for the distinction between key and key parts). For entities with a single key part (like component) this is no different than giving the key part value, but for entities with multiple key parts it means that a single value can be used to index. If key is given, no other keyword arguments are accepted.
The component_name keyword argument. This keyword argument is accepted because its name matches one of the key parts (and the only one).
Since components have only a single key part, the value for this single key part can also be given as a single positional parameter.

Incorrect indexing results in errors raised even before trying to hit the database:

>>> txn.component(key="lmc-controller", component_name="lmc-subarray-01")
Traceback (most recent call last):
ValueError: 'key' cannot be combined with other keyword arguments
>>> txn.component(wrong_key_part="lmc-controller")
Traceback (most recent call last):
ValueError: "wrong_key_part" is not a valid key part under /component
>>> txn.component("lmc-controller", "lmc-subarray-01")
Traceback (most recent call last):
ValueError: Only single positional argument can be given

Note also that providing key or key part values that don’t adhere to the key part patterns supported by the entity also results in errors:

>>> txn.component(key="invalid&char")
Traceback (most recent call last):
ska_sdp_config.operations.entity_operations.InvalidKey: Invalid key='invalid&char' for pattern=re.compile('^(?P<component_name>[a-zA-Z0-9_-]+)$')
>>> txn.component(component_name="invalid&char")
Traceback (most recent call last):
ValueError: "invalid&char" is not a valid value for key part "component_name" under /component
>>> txn.component("invalid&char")
Traceback (most recent call last):
ValueError: "invalid&char" is not a valid value for key part "component_name" under /component

See the API documentation to see all patterns in detail.

Entity operations

Once an entity has been indexed into, all its operations become available:

The key, key_parts and path attributes inform users of the key, key parts (names and values) and database path for the particular entity.
The create(), get(), update() and delete() methods allow users to perform basic CRUD operations.
The create_or_update() and exists() methods are small utilities built on top of the basic methods.

Additionally, if an entity type might give access to two extra attributes:

ownership is available on entity types that have an owner (that is, a process in the system that is in charge of them). It offers the basic take() and get() methods, together with the utility is_owned() and is_owned_by_this_process() ones. Additionally, the is_alive() and take_ownership_if_not_alive() methods are also provided.
state is available on entity types that have an associated state. It offers the basic create(), update() and get() methods.

`pydantic` entities

Indexing

For entities that are modelled via pydantic, manual indexing with strings as shown above is still possible, but not encouraged. Instead, high-level functions are provided that take a key, or a value, or either, and they internally perform the necessary indexing. See Entity operations (below) for more information.

To enforce this expected API usage when performing manual indexing, users must use the index_by_key_parts() method explicitly instead of using the magic __call__ function:

>>> from ska_sdp_config.entity import Deployment
>>> txn.deployment.index_by_key_parts(key="my-deployment")
<EntityOperations path="/deploy/my-deployment">

Entity operations

If an entity is modelled using pydantic then all of the operations listed above for dict entities are also available directly at the corresponding Transaction attribute. and require a entity value (or key) as argument. For writing operations it is convenient to specify the entity itself as argument, since the full value needs to be given anyway. For the rest, only the key parts from the entity are extracted and used, with the rest ignored, and in those cases the methods can accept the key as argument instead.

For example, the following shows the different indexing methods with the deployment entity:

>>> from ska_sdp_config.entity import Deployment
>>> deployment1 = Deployment(key="my-deployment1", kind="helm", args={})
>>> deployment2 = Deployment(key="my-deployment2", kind="helm", args={})

# Indexing happens as part of the operation
# Here we can index by key or by value to check if the deployment exists
>>> txn.deployment.exists("my-deployment1")
False
>>> txn.deployment.exists(deployment2)
False

# Writing operations require a full value; indexing still happens by key only
>>> txn.deployment.create(deployment1)
>>> txn.deployment.create(deployment2)

# Check both were created
>>> len(txn.deployment.list_keys())
2

# Reading via top-level get() can be indexed by value or by key
>>> all(txn.deployment.get(dpl) for dpl in (deployment1, deployment2))
True
>>> all(txn.deployment.get(key) for key in ("my-deployment1", "my-deployment2"))
True

Collective operations

Other operations are performed over the entire entity type. As such, these operations are directly available on the corresponding Transaction attribute. The only operations currently supported over a whole entity type is querying.

To query entities of a given type, users invoke the query_keys(), list_keys(), query_values() or list_values() methods on the corresponding attribute. By default they all return all keys/values for the given entity type. Constrains can be given in the form of keyword arguments that must match the entity’s key parts.

For entities that are modelled with pydantic, the returned keys and values are instances of the corresponding model. If entities are modelled with plain dictionaries, keys are strings and values are dictionaries.

For example, this code queries all subarray keys:

>>> txn.component.list_keys()
[]
>>> txn.component("lmc-controller").create({"value": 1})
>>> txn.component("lmc-subarray-01").create({"value": 2})
>>> sorted(txn.component.list_keys())
['lmc-controller', 'lmc-subarray-01']

If we wanted keys and values for all components whose IDs start with lmc we’d write:

>>> txn.component.list_values(component_name_prefix="lmc")
[('lmc-controller', {'value': 1}), ('lmc-subarray-01', {'value': 2})]

The next example creates a whole set of scripts, which are modelled with pydantic, then iterates over the first 5 vis-receive scripts with version 4.2.*:

from ska_sdp_config.entity import Script
from itertools import islice, product

# Create loads of scripts
names = ("vis-receive", "test-recv-addresses")
for name, major, minor, patch in product(names, range(10), range(10), range(10)):
   full_version = f"{major}.{minor}.{patch}"
   key = Script.Key(name=name, version=full_version, kind="realtime")
   image = f"{name}:{full_version}"
   sdp_version = "==0.21.0"
   script = Script(key=key, image=image, sdp_version=sdp_version)
   txn.script.create(script)
print(f"There are {len(txn.script.list_keys())} scripts")

# Query only some
print(f"First five vis-receive scripts with version 4.2.* follow:")
lazy_query = txn.script.query_values(name="vis-receive", version_prefix="4.2.")
for key, value in islice(lazy_query, 5):
   print(f"{key=}, {value=}")

There are 2000 scripts
First five vis-receive scripts with version 4.2.* follow:
key=Key(kind='realtime', name='vis-receive', version='4.2.0'), value=Script(key=Key(kind='realtime', name='vis-receive', version='4.2.0'), image='vis-receive:4.2.0', parameters={}, sdp_version='==0.21.0')
key=Key(kind='realtime', name='vis-receive', version='4.2.1'), value=Script(key=Key(kind='realtime', name='vis-receive', version='4.2.1'), image='vis-receive:4.2.1', parameters={}, sdp_version='==0.21.0')
key=Key(kind='realtime', name='vis-receive', version='4.2.2'), value=Script(key=Key(kind='realtime', name='vis-receive', version='4.2.2'), image='vis-receive:4.2.2', parameters={}, sdp_version='==0.21.0')
key=Key(kind='realtime', name='vis-receive', version='4.2.3'), value=Script(key=Key(kind='realtime', name='vis-receive', version='4.2.3'), image='vis-receive:4.2.3', parameters={}, sdp_version='==0.21.0')
key=Key(kind='realtime', name='vis-receive', version='4.2.4'), value=Script(key=Key(kind='realtime', name='vis-receive', version='4.2.4'), image='vis-receive:4.2.4', parameters={}, sdp_version='==0.21.0')

Supported entities

The table below summarises the supported entities and their operations. Note that for those entities modelled via pydantic there is a model class, while for plain dict entities there are key parts.

Entities supported by `Transaction`
Attribute	Prefix/path	Model class	Key parts	Has `ownership`?	Has `state`?
`component`	`/component`	–	`component_name`	`True`	`False`
`deployment`	`/deploy`	`Deployment`	–	`False`	`True`
`execution_block`	`/eb`	`ExecutionBlock`	`eb_id`	`False`	`True`
`processing_block`	`/pb`	`ProcessingBlock`	–	`True`	`True`
`script`	`/script`	`Script`	–	`False`	`False`
`flow`	`/flow`	`Flow`	–	`True`	`True`

Arbitrary path access

On top of the entity-specific access provided by Transaction, there is also an extra arbitrary attribute that allows operating over any arbitrary path in the database. This is useful for tools performing operations over generic paths, or to quickly experiment with storing data under certain paths before committing to adding a new entity in this package.

To access a particular path users need to index into arbitrary as described in Indexing, although here a single positional argument indicates the path. The result of that call is the set of operations described in Entity operations.

The following example uses arbitrary path access to first write into "/my_path", then obtain the value that has just been written:

with warnings.catch_warnings(record=True) as warns:
    txn.arbitrary("/my_path").create({"my_value": 3})
    print(txn.arbitrary("/my_path").get())
print(f"{len(warns)} warnings generated")

{'my_value': 3}
2 warnings generated

The example above is accessing /my_path, which is not a path that is handled by any of the supported entities. In those cases, a warning is raised so that users are aware of this unofficial access.

Note

This arbitrary path support shouldn’t be abused as a long-term solution if the goal is to introduce a new entity to the system. New entities should be explicitly added to this package and used instead.

Accessing known paths raises no warnings:

txn.component("lmc-subarray-01").create({"value": 1})
with warnings.catch_warnings(record=True) as warns:
    print(f'Subarray: {txn.arbitrary("/component/lmc-subarray-01").get()}')
print(f"{len(warns)} warnings generated")

Subarray: {'value': 1}
0 warnings generated

Entities and Operations

Overview

Entity models

Operating on entities

The self attribute

dict entities

Indexing

Entity operations

pydantic entities

Indexing

Entity operations

Collective operations

Supported entities

Arbitrary path access

The `self` attribute

`dict` entities

`pydantic` entities