Migrating to 1.4

This migration guide lists all the deprecations and unintentional breaking changes introduced by ska-tango-base release 1.4.0. Despite the breaking changes, in the majority of cases developers should still be able to update from version 1.x without having to make any changes. However, there are two features that have been deprecated with this release and where ever possible deprecation warnings are emitted whenever they are used. This guide contains recommendations about how to resolve the deprecation warnings in order to prepare for the major release of ska-tango-base where these will be removed.

There are two unintentional breaking changes that affect the following sets of devices:

• Any devices overriding base class Tango attributes (such as adminMode) and calling the “super” version of the attribute via the super class.

• Any devices manually pushing events for base class Tango attributes rather than using the e.g. _update_admin_mode() method.

Unfortunately, it is not possible to fix these breaking changes in a patch release. This guide contains information about these breaking changes and provides instructions on how to resolve them.

In addition to the feature deprecations, the module structure of ska-tango-base has been re-organised for the 1.4.0 release. This re-organisation is backwards compatible, however, there have been some deprecations. Also, a new type_hints module is provided and for the LrcSubscriptions and CommandTracker classes protocol types have been added to use instead of the concrete type for type hint purposes. These protocols are each the structural interface that the corresponding class will support in the next major release of ska-tango-base. This guide describes how to write ska-tango-base import statements that are future-proof and how to use the provided protocol classes to ensure that your usage of the LrcSubscriptions and CommandTracker is future proof.

Warning

The SKABaseDevice now makes use of the always_executed_hook() and delete_device() methods. Subclasses of SKABaseDevice must always ensure they call the super() version of these methods if they override them otherwise SKABaseDevice will fail to clean up after itself. This has always been a requirement for these functions, however, it may have worked in previous releases as the methods were not used in ska-tango-base.

ExecutePendingOperations deprecated

Previous releases of ska-tango-base allowed subclasses of SKABaseDevice to call so-called “Tango operations”, such as tango.LatestDeviceImpl.push_change_event() from any thread even though Tango requires these operations be called from an omnithread. This was achieved by not calling the operations directly, but scheduling the operations to be called at some point later by a polled ExecutePendingOperations() command.

In the 1.4.0 release of ska-tango-base the ExecutePendingOperations() command has been removed. It is currently still possible to call tango operations from a non-omnithread as now the operation will be scheduled to be executed on the new signal bus object. However, this is now deprecated and a warning will be emitted when a tango operation is scheduled for execution on the signal bus object.

Rationale

As the ExecutePendingOperations() command is polled at 100ms it adds, on average, a 50ms delay in pushing an event from the device. This undermines the goal of having an event driven Tango device. Due to this, ExecutePendingOperations() was removed in favour of the signal bus object.

Supporting the calling of Tango operations from non-omnithreads leads to subtle differences in behaviour for functions which call these operations, depending on the context where they are being called. This can lead to difficult to track down bugs, specifically around locking in a multi-threaded application. It is more straight forward to deal with these issues if developers explicitly opt-in to having code running on a separate thread by using the signal bus directly.

Preparing for removal

All threads which wish to invoked Tango operations should use the tango.EnsureOmniThread context manager to make the thread an omnithread. This should be done exactly once, i.e. the with statement should wrap the entire function that runs on the thread. This includes threads spawned by a component manager if the thread calls callbacks from the Tango device which ends up calling push_change_event().

The Poller class has been updated to ensure that the polling thread is an omni-thread. Similarly, the TaskExecutor has been updated to use the tango.utils.PyTangoThreadPoolExecutor when using pytango>=10.0.0, which uses omnithreads. If running against an older version of pytango, calling tango operations from a task scheduled with the TaskExecutor will emit a deprecation warning unless the call is wrapped in with tango.EnsureOmniThread(). However, note that calling tango.EnsureOmniThread multiple times in the same thread is bad practice, so to avoid these warnings you should aim to update your pytango dependency to version 10.0.0 or later.

For example, the following component manager uses an omnithread because its Tango device has overridden ska_tango_base.base.base_device.SKABaseDevice._component_state_changed() so that _update_component_state() ends up calling push_change_event().

import ska_tango_base as stb
...

class MyComponentManager(stb.base.BaseComponentManager):
  def __init__(self, *args, **kwargs):
    super().__init__(*args, **kwargs)
    self.thread = threading.Thread(target=self.run)
    self.thread.start()

  def run(self):
    with tango.EnsureOmniThread():
      ...
      self._update_component_state(qux="foo")

Note

The default ska_tango_base.base.base_device.SKABaseDevice._component_state_changed() and ska_tango_base.base.base_device.SKABaseDevice._communication_state_changed() callbacks no longer call Tango operations, instead they schedule them to be called by the signal bus at some point later. However, if your Tango device is overriding ska_tango_base.base.base_device.SKABaseDevice._component_state_changed() it may be calling tango.LatestDeviceImpl.push_change_event() directly.

Alternatively, you could ensure that a thread does not need to call any Tango operations using the signal bus object. For example, you can use attribute_from_signal() to automatically push events from a separate thread whenever a value is emitted for the Signal() data descriptor on the component manager:

import ska_tango_base as stb
...

class MyComponentManager(stb.software_bus.SharingObserver, stb.BaseComponentManager):
  qux = stb.software_bus.Signal[int]()

  def __init__(self, *args, **kwargs):
    super().__init__(*args, **kwargs)
    self.thread = threading.Thread(target=self.run)
    self.thread.start()

  def run(self):
    ...
    self.qux = 1

class MyDevice(stb.SKABaseDevice[MyComponentManager]):
   quxAttr = stb.software_bus.attribute_from_signal("_component_manager.qux", dtype=int)
   ...

SKA command objects deprecated

The ska-tango-base release 1.4.0 has deprecated the SKA command objects for implementing Tango commands. Specifically, the register_command_object() and get_command_object() functions are deprecated and will emit a warning when called.

In order maintain backwards compatibility the default implementations of most standard commands still use the command objects and will emit a deprecation warning if the command has not been overridden via a new mechanism. In general, teams are required to override the execute_<cmd>() methods to provide implementations which don’t use the command objects to address this deprecation warning.

The exception to this is the Abort() command which will not emit a deprecation warning unless overridden via the command object mechanism. To modify the behaviour of the Abort() command, teams should override the schedule_abort_task() method.

As a corollary, the InitCommand command object has been deprecated and will emit a warning if used. In order to acknowledge the deprecation, set InitCommand = None for your device and override the init_device() method to change initialisation behaviour.

Your init_device() must call init_completed() once initialisation has been completed. This may happen asynchronously if you have a long running initialisation process. See Initialising your SKA Tango device for more details.

Rationale

The SKA command objects introduce a lot of boilerplate in order to implement a command. More over, the current implementation does not handle managing the is_<cmd>_allowed() methods correctly for long running commands, nor does it handle transitioning the ObsState correctly for the ska_tango_base.subarray.subarray_device.SKASubarray when using long running commands.

Fixing these issues requires a breaking change with regards to how commands are implemented, so we took the opportunity to simplify the API and reduce the amount of boiler plate required when using ska-tango-base.

Preparing for removal

Standard SKA state transition commands

The behaviour of standard state transition commands should now be implemented by overriding the execute_<cmd>() methods. For so-called fast commands, this is simply a case of moving the logic from your command object into the execute_<cmd>() method. For long running commands there is a little more work to do. The most straight forward way to migrate a default SubmittedSlowCommand is to use the new allocate_lrc() and convert_submission_result_to_lrc_return() methods to call the component manager method that submits the task directly. For example, the following execute_On() will have the same behaviour as the default SubmittedSlowCommand object:

import ska_tango_base as stb
...

class MyDevice(stb.SKABaseDevice[MyComponentManager]):
   ...
   @stb.long_running_commands.mark_long_running
   def execute_On(self) -> stb.type_hints.DevVarLongStringArrayType:
       command_id, task_callback = self.allocate_lrc("On")
       # WARNING: Unless handled by the component manager, this will not call
       # the is allowed method when the task is dequeued
       status, message = self.component_manager.on(task_callback)
       return self.convert_submission_result_to_lrc_return(command_id, status, message)

The @mark_long_running decorator here tells the default is_On_allowed() method that we should not check the state machine when the Tango command is initially called. If the command is not long running, then this decorator should not be used, nor should we allocate a long running command with the call to allocate_lrc().

Although the implementation above of execute_On() will not, incorrectly, check the state machine when the command is submitted, it will also not, correctly, check the state machine when the task is about to be executed, unless the component manager’s on() method explicitly does this. This can be remedied by pulling the task submission machinery from the component manager into the Tango device using submit_lrc_task():

import ska_tango_base as stb
...

class MyComponentManager(
    stb.executor.TaskExecutorComponentManager, stb.base.BaseComponentManager
):
    ...

    def do_on(
        self,
        task_callback: stb.type_hints.TaskCallbackType,
        task_abort_event: threading.Event,
    ) -> None:
       ...

    # Now unused, as we are submitting the task from the Tango device
    def on(
        self, task_callback: stb.type_hints.TaskCallbackType | None = None
    ) -> tuple[scm.TaskStatus, str]:
        return self._task_executor.submit(self.do_on, task_callback=task_callback)

class MyDevice(stb.SKABaseDevice[MyComponentManager]):
   ...
   @stb.long_running_commands.mark_long_running
   def execute_On(self) -> stb.type_hints.DevVarLongStringArrayType:
       command_id, task_callback = self.allocate_lrc("On")
       status, message = self.submit_lrc_task("On", self.component_manager.do_on, task_callback)
       return self.convert_submission_result_to_lrc_return(command_id, status, message)

As this is still quite a lot of boiler plate, ska-tango-base provides the @submit_lrc_task decorator, which does the same as above:

import ska_tango_base as stb
...

class MyDevice(stb.SKABaseDevice[MyComponentManager]):
   ...
   @stb.long_running_commands.submit_lrc_task
   def execute_On(self) -> stb.type_hints.TaskFunctionType:
       return self.component_manager.do_on

Here we are returning the component manager method to be scheduled as a long running command task. If our command accepts arguments, we need to wrap the component manager method in a closure to store the arguments. The decorator @validate_json_args can be used to create a method that accepts a single JSON string from a method that accepts multiple keyword arguments. For example, the following overrides the behaviour of the AssignResources() command, which accepts its resources arguments in a JSON encoded object:

class MyDevice(stb.SKASubarray[MyComponentManager]):
    ...

    AssignResources_SCHEMA: dict[str, stb.type_hints.JSONData] = {
        "$schema": "https://json-schema.org/draft/2020-12/schema",
        "$id": "https://skao.int/ska-tango-base/ReferenceSkaSubarray_AssignResources.json",
        "title": "ska-tango-base ReferenceSkaSubarray AssignResources schema",
        "description": "Schema for ska-tango-base ReferenceSkaSubarray AssignResources command",
        "type": "object",
        "properties": {
            "resources": {
                "description": "Resources to assign",
                "type": "array",
                "items": {"type": "string"},
            },
        },
        "required": ["resources"],
    }

    @stb.long_running_commands.submit_lrc_task
    @stb.validators.validate_json_args
    def execute_AssignResources(
        self, resources: list[str]
    ) -> stb.type_hints.TaskFunctionType:
        def task(
            task_callback: stb.type_hints.TaskCallbackType,
            task_abort_event: threading.Event,
        ) -> None:
            self.component_manager.do_assign(
                set(resources),
                task_callback=task_callback,
                task_abort_event=task_abort_event,
            )

        return task

Alternatively, for the On(), Off(), Standby() and Reset() commands you can remove these from the Tango interface when using SKABaseDevice or one of its subclasses by assigning None to them at the class scope. For example, the following device disables its Reset() command:

import ska_tango_base as stb
...

class MyDevice(stb.SKASubarray[MyComponentManager]):
    Reset = None
    ...

InitCommand command

To acknowledge the InitCommand deprecation, users of SKABaseDevice must assign InitCommand to None at class scope and provide an override for init_device(). The init_device() override must call init_completed() once initialisation has finished. For example, the following device does the minimum required to acknowledge the deprecation and suppress the warning:

import ska_tango_base as stb
...

class MyDevice(stb.SKASubarray[MyComponentManager]):
    InitCommand = None

    def init_device(self) -> None:
        super().init_device()
        self.init_completed()

Warning

When migrating your InitCommand.do() method, note that you will be asking the Tango device as self._device, however, in the init_device() method the Tango device is just self.

Abort command

A deprecation warning will be emitted for subclasses that are overriding the Abort() command using the command object mechanism. Classes wishing to modify the behaviour of abort should now override the schedule_abort_task() method. This method receives a task_callback() constructed via allocate_lrc() and must start the abort process asynchronously and return TaskStatus.IN_PROGRESS immediately. The abort process must include a call to self.task_executor.abort.

When using SKABaseDevice the default implementation of schedule_abort_task() will call the component manager’s abort() method. Typically, in this case, all that is required to change the behaviour of the Abort() command is to override this method on the component manager.

User-defined Tango commands

For user-defined commands, the @long_running_command decorator can be used instead of @tango.server.command() to create a long running command from a task. For example, the following device defines the long running command Qux():

import ska_tango_base as stb

class MyDevice(SKABaseDevice[MyComponentManager]):
  ...
  Qux_SCHEMA: dict[str, JSONData] = {
      "$schema": "https://json-schema.org/draft/2020-12/schema",
      "$id": "artefact.skao.int/mylrc.schema.json",
      "title": "Qux schema",
      "description": "Validates the keyword arguments",
      "type": "object",
      "properties": {
          "foo": {"type": "string"},
          "bar": {"type": "number"},
      },
  }

  @stb.long_running_commands.long_running_command
  @stb.validators.validate_json_arg
  def Qux(self, foo: str, bar: int) -> TaskFunctionType:
     def task(task_callback, task_abort_event) -> int:
       return self.component_manager.do_qux(
         foo,
         bar,
         task_callback=task_callback,
         task_abort_event=task_abort_event
       )

     return task

When using the SubarrayInterface or its child SKASubarray, the obs_state_model actions to transition the ObsState are performed automatically at the start and end of each command when using the submit_lrc_task()/long_running_command() decorators.

Note

The component-oriented actions must still be called explicitly at the appropriate time in the execute_<cmd>() implementations if a component manager is not used to update component states.

The following device does not use a component manager and overrides the AssignResources() command, using the @TaskExecutor.task decorator to simplify the task implementation. The task calls component_resourced() to indicate to the obs state machine that the AssignResources() command successfully assigned resources:

import ska_tango_base as stb
import ska_control_model as scm

class MyDevice(stb.subarray.SubarrayInterface):
  ...

  AssignResources_SCHEMA: dict[str, stb.type_hints.JSONData] = {
      "$schema": "https://json-schema.org/draft/2020-12/schema",
      "$id": "https://skao.int/ska-tango-base/ReferenceSkaSubarray_AssignResources.json",
      "title": "ska-tango-base ReferenceSkaSubarray AssignResources schema",
      "description": "Schema for ska-tango-base ReferenceSkaSubarray AssignResources command",
      "type": "object",
      "properties": {
          "resources": {
              "description": "Resources to assign",
              "type": "array",
              "items": {"type": "string"},
          },
      },
      "required": ["resources"],
  }

  @stb.long_running_commands.submit_lrc_task
  @stb.validators.validate_json_args
  def execute_AssignResources(self, resources: list[str]) -> stb.type_hints.TaskFunctionType:

      @stb.executor.TaskExecutor.task
      def task(
          progress_callback: stb.type_hintsProgressCallbackType, task_abort_event: threading.Event
      ) -> tuple[ResultCode, str]:
          ...
          self.component_resourced()
          return scm.ResultCode.OK, "AssignResources successful"

      return task

Module restructuring

The 1.4.0 release has reorganised the module structure of ska-tango-base. All previously valid import statements are still valid, however, some modules have been deprecated and will emit a deprecation warning when imported. In addition, importing some objects from particular modules is deprecated and will emit a deprecation warning.

An overview of all the deprecated modules can be found in Deprecate Modules. Each deprecation warning provides directions as to where the object can now be found. In some cases, the object has a new name and this will be explained in the deprecation warning.

In the case of CommandTracker and LrcSubscriptions the deprecation warning only provides protocols as alternatives. These protocols are intended to be used in type annotations and are designed to hide the concrete type that is created by ska-tango-base.

In a future release, the concrete types will become private and changes to them will not be considered breaking.

Note

The public CommandTracker concrete implementation is, in fact, no longer used internally for this release of ska-tango-base. Instead there is already an alternative private _CommandTracker implementation that is being used instead. This new implementation models the same CommandTrackerProtocol, however, it has different construction requirements so the original CommandTracker is provided for compatibility, in case teams were using the concrete object themselves.

In the case of the LrcSubscriptions class, we now use a new _LRCSubscriptions name internally, but provide the LrcSubscriptions name as an alias for backwards compatibility.

Rationale

As the ska-tango-base package has evolved the organisation of modules has become somewhat messy and the restructuring is an effort to improve this.

The use of protocols for CommandTracker and LrcSubscriptions is motivated by the desire to avoid excess coupling between ska-tango-base and users of ska-tango-base and to express more clearly what use cases of the object are intended to be supported going forwards. This, in turn, allows the evolution of ska-tango-base in the future without breaking users.

Preparing for removal

To prepare for the removal of the public CommandTracker and LrcSubscriptions update type annotations to use the protocols ska_tango_base.type_hints.CommandTrackerProtocol and ska_tango_base.type_hints.LRCSubscriptionsProtocol respectively.

If you have a use case other than a type annotation, please get in touch with the maintainers of ska-tango-base and we can provide a mechanism to support your use case.

Extending attributes

The 1.4.0 release introduced a prescribed way to change the behaviour of most standard attributes provided by the base classes. The attributes which allow this are namely:

• BaseInterface.adminMode

• BaseInterface.commandedState

• BaseInterface.healthState

• SKABaseDevice.controlMode

• SKABaseDevice.simulationMode

• SKABaseDevice.testMode

• ControllerInterface.availableCapabilities

• ControllerInterface.maxCapabilities

• ObsInterface.obsState

• ObsInterface.commandedObsState

• ObsInterface.obsMode

• ObsInterface.configurationProgress

• ObsInterface.configurationDelayExpected

• SubarrayInterface.activationTime

Before 1.4.0, it was possible to override standard attributes and call the “super” implementation by accessing the superclass’s version of the attribute. This pattern is uncommon and was never officially documented anywhere. For example:

import tango
import ska_tango_base as stb

class MyDevice(stb.SKABaseDevice):
    ...
    @tango.server.attribute
    def adminMode(self) -> AdminMode:
        """Only overriding in order to override the setter."""
        return stb.SKABaseDevice.adminMode.fget(self)

    @adminMode.write
    def adminMode(self, value: AdminMode) -> None:
        """Set the Admin Mode of the device."""
        stb.SKABaseDevice.adminMode.fset(self, value)
        # Do something else whenever the adminMode is changed
        ...

Doing the above will not work with release 1.4.0 for technical reasons we will not cover here. As such, it is considered an unintentional breaking change. Fortunately, release 1.4.0 introduced a simpler, official mechanism to extend the behaviour of these attributes by overriding their corresponding read_<attribute>() or, if provided, write_<attribute>() methods. As this is standard method overriding, the usual super().read_<attribute>() method is available. For example, to achieve the above behaviour with 1.4.0:

import ska_tango_base as stb

class MyDevice(stb.SKABaseDevice):
    ...
    def write_adminMode(self, value: AdminMode) -> None:
        """Set the Admin Mode of the device."""
        super().write_adminMode(value)
        # Do something else whenever the adminMode is changed
        ...

Note

If you need to completely override an attribute with a custom implementation, please get in touch with the maintainers of ska-tango-base so we can consider supporting your use case.

Manually pushing attributes events

Since before the 1.4.0 release, the SKABaseDevice has provided several _update_<x>() methods for the subclass to call whenever they wish to update the corresponding attribute. Notably, these are:

• BaseInterface._update_admin_mode

• BaseInterface._update_state

• SKABaseDevice._update_health_state

• ObsInterface._update_obs_state

Each of these methods updates the corresponding python attribute and pushes Tango events. Although it was intended that subclasses would call these functions to update the attributes, they unfortunately never appeared in the public documentation until release 1.4.2. In the 1.4.0 release, we changed how events were pushed for these attributes so that writing to the corresponding python attributes e.g. BaseInterface._health_state results in events being pushed automatically. Unfortunately, this change means that events can end up being pushed multiple times after updating to ska-tango-base 1.4.0 in the following scenario:

1. The device does not use the _update_<x>() method and instead pushes events itself manually.

2. The device calls set_change_event()/set_archive_event() itself with detect=False.

If neither of the above is true, then events will still be pushed normally. If your device satisfies both the above conditions then you can resolve the issue by updating to use the _update_<x>() method.