.. _lrc-task-guidelines: ======================================= How to implement a long running command ======================================= The SKA long running command (LRC) machinery allows clients to schedule long running tasks on an SKA Tango device and receive updates on the tasks progress using the `LRC protocol `_. In order to support this an SKA Tango device needs to provide, for each long running command it implements, a task to be scheduled and a regular Tango command to do the scheduling. This guide explains the recommended way to implement an LRC using ska-tango-base >=1.4.0, with or without using a component manager. This guide is for users implementing new devices. Please refer to the :ref:`1.4 migration guide ` on how to migrate existing devices that are using deprecated SKA command objects. .. contents:: Steps :depth: 2 :local: :backlinks: none Step 1: Create a task to fulfil the LRC ======================================= The first step to implement an LRC is to create a task method which performs the resource-intensive/blocking work of the command. The task should report its status/progress via a callback and periodically check whether it has been requested to abort. It is recommended to use the :meth:`TaskExecutor.task() ` decorator to apply some of this boilerplate code for you, but it is also possible to manage the task status yourself if needed. .. tip:: A task can be a method on the Tango device class or another object such as a component manager, or even a free function. Using the task decorator ------------------------ The :meth:`TaskExecutor.task() ` decorator wraps a method that accepts a progress callback and abort event with common task executor boilerplate code to transition the task through the :ref:`task state machine `. The decorator will pass the progress callback and abort event as keyword arguments with the names :obj:`!progress_callback` and :obj:`!task_abort_event`. Additional arguments beyond these are supported and will be passed through to the wrapped function by the decorator. The method should also return the result of the task as encodable :class:`~ska_tango_base.type_hints.JSONData` - some recommendations follow below. Partial example: .. code:: python import ska_tango_base as stb import ska_control_model as scm ... @stb.executor.TaskExecutor.task def very_slow_task( ... progress_callback: stb.type_hints.ProgressCallbackType, task_abort_event: threading.Event | None = None, ) -> stb.type_hints.JSONData: ... return ([scm.ResultCode.OK], ["Task completed"]) Task dos and don'ts ------------------- There are a few things a task method should do to ensure it behaves correctly as an LRC task: .. admonition:: Important guidelines **Reporting progress** The task should regularly call the :obj:`!progress_callback` to report the progress of the task. The progress is expected to be an integer, and it is recommended to use this integer to represent a percentage from 0 to 100. How to interpret the task progress should be well documented for clients invoking the LRC. **Task result** The task result is the return value of the task method decorated with :meth:`TaskExecutor.task() `. It is recommended to always include a :class:`~ska_control_model.ResultCode` to indicate to clients if the task has completed successfully or not. Ideally, this :class:`~ska_control_model.ResultCode` should be accessed with ``result[0]`` to fit in with task results provided by ska-tango-base. A client should know the type of ``result[1]`` based on the value of ``result[0]``. If your task can complete "partially successfully", consider using multiple result codes to provide more details. For example, if your task coordinates multiple subordinate devices, you might provide a result such as the following: .. code:: python (scm.ResultCode.OK, { "total_success": False, "device_responses":[ (scm.ResultCode.OK, "OK"), (scm.ResultCode.FAILED, "Not enough quux available"), ... ] }) **Aborting** The task should check at regular intervals whether the abort event has been set. If :meth:`!task_abort_event.is_set()` returns ``True``, the task should quickly perform only essential cleanup and raise :class:`~ska_tango_base.executor.executor.TaskAborted`. **Handling exceptions** Do not raise exceptions (other than :class:`~ska_tango_base.executor.executor.TaskAborted`) from your task method to indicate normal task failures. Instead, use the :class:`~ska_control_model.TaskStatus` and :class:`~ska_control_model.ResultCode` to communicate the outcome of the task. Raising exceptions should be reserved for abnormal failures (i.e. bugs). Managing the task status yourself --------------------------------- If more flexibility is required for your task, you could manage task status yourself using the :meth:`!task_callback` instead of using the :meth:`TaskExecutor.task() ` decorator. However, care must be taken when managing the task status manually to ensure that the task status follows the state machine as defined by the LRC protocol. When writing a task without the :meth:`TaskExecutor.task() ` decorator, the task should return ``None`` and the task result should be reported via the :meth:`!task_callback`. Briefly, the task status state machine requires the following. At the start of your task method the task status must be updated to :obj:`TaskStatus.IN_PROGRESS ` via the :meth:`!task_callback`. During the execution of your task you should update the task progress also via the :meth:`!task_callback` and periodically check the abort event as explained above. Before your task method returns it must update the task status to be either :obj:`TaskStatus.COMPLETED ` or :obj:`TaskStatus.ABORTED ` as appropriate, and provide a task result via the :meth:`!task_callback`. See :ref:`lrc-concept-tasks` for details about the task status state machine. .. code:: python import ska_tango_base as stb import ska_control_model as scm ... def very_slow_task( ... task_callback: collections.abc.Callable, task_abort_event: threading.Event, ) -> None: # Indicate that the task has started task_callback(status=scm.TaskStatus.IN_PROGRESS) while/for ...: # Update the task progress task_callback(progress=...) # Do something that takes long to complete ... # Periodically check that tasks have not been ABORTED if task_abort_event.is_set(): # Indicate that the task has been aborted task_callback( status=scm.TaskStatus.ABORTED, result=(scm.ResultCode.ABORTED, "Task aborted"), ) return # Indicate that the task has completed task_callback( status=scm.TaskStatus.COMPLETED, result=(scm.ResultCode.OK, "Task completed"), ) Step 2: Decide on a concurrency mechanism ========================================= Once you have a task to be scheduled you need to decide on where to schedule it, i.e. what concurrency mechanism to use. This is a decision that needs to be taken for the SKA Tango device as a whole rather than for each long running command. The long running command infrastructure provided by ska-tango-base requires you to provide an object modelling the :class:`~ska_tango_base.type_hints.TaskExecutorProtocol` as a concurrency mechanism. If you are working on a device inheriting from any of the base device classes that subclass :class:`~ska_tango_base.base.base_device.SKABaseDevice`, then the device will, by default, expect the :class:`~ska_tango_base.type_hints.TaskExecutorProtocol` object to be provided by the component manager. In order to fulfil this, the component manager must inherit from :class:`~ska_tango_base.executor.executor_component_manager.TaskExecutorComponentManager` which provides a :class:`~ska_tango_base.executor.executor.TaskExecutor` for queuing and executing asynchronous tasks. However, subclasses of :class:`~ska_tango_base.base.base_device.SKABaseDevice` can provide a :class:`~ska_tango_base.executor.executor.TaskExecutor` directly by inheriting from :class:`~ska_tango_base.long_running_commands.mixin.LRCMixin`, thereby not requiring a component manager specifically for LRCs (i.e. making it optional). It is possible to implement long running commands using a different concurrency mechanism. The :class:`~ska_tango_base.long_running_commands.mixin.LRCMixin` allows providing other objects that implement the :class:`~ska_tango_base.type_hints.TaskExecutorProtocol` by overriding :meth:`~ska_tango_base.long_running_commands.mixin.LRCMixin.create_task_executor`, but this is not covered in this guide. .. admonition:: Recommendation Always have your device inherit from :class:`~ska_tango_base.long_running_commands.mixin.LRCMixin`, unless your component manager needs access to the task executor for some reason - otherwise have your component manager inherit from :class:`~ska_tango_base.executor.executor_component_manager.TaskExecutorComponentManager`. Step 3: Implement a Tango command to schedule the task ====================================================== The next step is to implement a Tango command to submit the task to the :class:`~ska_tango_base.type_hints.TaskExecutorProtocol` object your device uses. This step differs slightly for a user-defined command versus so-called standard SKA commands already defined in the base classes. User-defined LRC ---------------- For a user-defined LRC use the :meth:`~ska_tango_base.long_running_commands.decorators.long_running_command` decorator to create a Tango command method which submits the task. For example a device using a component manager: .. code:: python import ska_tango_base as stb ... class MyDevice(stb.SKABaseDevice[MyComponentManager]): ... @stb.long_running_commands.long_running_command def VerySlow(self) -> stb.type_hints.TaskFunctionType: """Long running command.""" return self.component_manager.very_slow_task If you are working on a device constructed modularly with the :class:`~ska_tango_base.long_running_commands.mixin.LRCMixin`: .. code:: python import ska_tango_base as stb ... class MyDevice(stb.long_running_commands.LRCMixin, stb.SKADevice): .. @stb.long_running_commands.long_running_command def VerySlow(self) -> stb.type_hints.TaskFunctionType: """Long running command.""" return self.very_slow_task .. tip:: The :meth:`~ska_tango_base.long_running_commands.decorators.long_running_command` decorator accepts the same parameters as :meth:`!tango.command()`, so you can specify input argument types, descriptions, etc. as needed. Standard SKA command as an LRC ------------------------------ The :class:`~ska_tango_base.base.base_interface.BaseInterface` and :class:`~ska_tango_base.subarray.subarray_interface.SubarrayInterface` classes (which are base classes for :class:`~ska_tango_base.base.base_device.SKABaseDevice` and :class:`~ska_tango_base.subarray.subarray_device.SKASubarray` respectively) define a set of standard SKA commands which subclasses are expected to override via the :meth:`!execute_` methods. It is possible to make these long running commands using the :meth:`~ska_tango_base.long_running_commands.mixin.AbstractLRCMixin.submit_lrc_task` decorator. .. tip:: If your device is a subarray device and must also implement the standard subarray commands, you can inherit from both :class:`~ska_tango_base.subarray.subarray_component_manager.SubarrayComponentManager` and :class:`~ska_tango_base.executor.executor_component_manager.TaskExecutorComponentManager`. .. code:: python import ska_tango_base as stb ... class MySubarrayComponentManager( stb.subarray.SubarrayComponentManager, stb.executor.TaskExecutorComponentManager ): """A subarray component manager.""" ... When required to be an LRC, the behaviour of a standard SKA command should be implemented in a task that is returned by the relevant :meth:`!execute_` method. The :meth:`!execute_` method should then be decorated with the :meth:`~ska_tango_base.long_running_commands.mixin.AbstractLRCMixin.submit_lrc_task` decorator. For example, here we are returning the component manager method to be submitted as a long running :meth:`!On` command task: .. code:: python import ska_tango_base as stb ... class MyComponentManager(stb.executor.TaskExecutorComponentManager): ... def do_on( self, task_callback: stb.type_hints.TaskCallbackType, task_abort_event: threading.Event, ) -> None: # Implement the long running task here ... return class MyDevice(stb.SKABaseDevice[MyComponentManager]): ... @stb.long_running_commands.submit_lrc_task def execute_On(self) -> stb.type_hints.TaskFunctionType: return self.component_manager.do_on .. admonition:: Important notes **Optional commands** The standard SKA commands provided by :class:`~ska_tango_base.base.base_interface.BaseInterface` are optional, namely :func:`!On`, :func:`!Off`, :func:`!Standby` and :func:`!Reset`, and will only be included in the Tango interface if the corresponding :func:`!execute_` methods are overridden. However, for backwards compatibility the :class:`~ska_tango_base.base.base_device.SKABaseDevice` will always include these commands in the interface. A subclass of :class:`~ska_tango_base.base.base_device.SKABaseDevice` can disable the commands by assigning them to ``None``. For example: .. code:: python import ska_tango_base as stb ... class MyDevice(stb.SKASubarray[MyComponentManager]): Reset = None ... **ObsState management** When using the :class:`~ska_tango_base.subarray.subarray_interface.SubarrayInterface` or its child :class:`~ska_tango_base.subarray.subarray_device.SKASubarray`, the :attr:`~ska_tango_base.subarray.subarray_interface.AbstractSubarrayInterface.obs_state_model` actions to transition the :class:`!ObsState` are performed automatically at the start and end of each command when using the :func:`~ska_tango_base.long_running_commands.decorators.submit_lrc_task` or :func:`~ska_tango_base.long_running_commands.decorators.long_running_command` decorators. However, the :meth:`!component_` actions must still be called explicitly at the appropriate time by the task implementing the LRC. When using a component manager, these :meth:`!component_` methods are automatically called when :meth:`~ska_tango_base.base.base_component_manager.BaseComponentManager._update_component_state` is called with the appropriate arguments. The following device does not use a component manager and overrides the :func:`!AssignResources` command, using the :func:`TaskExecutor.task ` decorator to simplify the task implementation. The task calls :func:`~ska_tango_base.subarray.subarray_interface.AbstractSubarrayInterface.component_resourced` to indicate to the observation state machine that the :func:`!AssignResources` command successfully assigned resources: .. code:: python import ska_tango_base as stb import ska_control_model as scm ... class MyDevice(stb.subarray.SubarrayInterface): ... AssignResources_SCHEMA: dict[str, stb.type_hints.JSONData] = { "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://skao.int/ska-tango-base/AssignResources.json", "title": "MyDevice AssignResources schema", "description": "Schema for MyDevice AssignResources command", "type": "object", "properties": { "resources": { "description": "Resources to assign", "type": "array", "items": {"type": "string"}, }, }, "required": ["resources"], } @stb.long_running_commands.submit_lrc_task @stb.validators.validate_json_args def execute_AssignResources( self, resources: list[str] ) -> stb.type_hints.TaskFunctionType: @stb.executor.TaskExecutor.task def task( progress_callback: stb.type_hintsProgressCallbackType, task_abort_event: threading.Event, ) -> tuple[ResultCode, str]: ... self.component_resourced() return [scm.ResultCode.OK], ["AssignResources successful"] return task **Customising standard SKA command behaviour** If you need more control over the behaviour of a standard SKA command, you can look at using the :meth:`~ska_tango_base.long_running_commands.decorators.mark_long_running` decorator and :meth:`~ska_tango_base.long_running_commands.mixin.AbstractLRCMixin.allocate_lrc` method. See the :ref:`1.4 migration guide ` for more details. Step 4: Handling command arguments ================================== If your command accepts arguments, you need to wrap your task method in a closure, or use :meth:`!functools.partial`, to capture the arguments to be used by the task. Additionally, the :func:`~ska_tango_base.validators.validate_json_args` decorator can be used to create a method that accepts a single JSON string from a method that accepts multiple keyword arguments. For example, the following overrides the behaviour of the :func:`!AssignResources` command, which accepts its :obj:`!resources` argument in a JSON encoded object: .. code:: python class MyDevice(stb.SKASubarray[MyComponentManager]): ... AssignResources_SCHEMA: dict[str, stb.type_hints.JSONData] = { "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://skao.int/ska-tango-base/ReferenceSkaSubarray_AssignResources.json", "title": "ska-tango-base ReferenceSkaSubarray AssignResources schema", "description": "Schema for ska-tango-base ReferenceSkaSubarray AssignResources command", "type": "object", "properties": { "resources": { "description": "Resources to assign", "type": "array", "items": {"type": "string"}, }, }, "required": ["resources"], } @stb.long_running_commands.submit_lrc_task @stb.validators.validate_json_args def execute_AssignResources( self, resources: list[str] ) -> stb.type_hints.TaskFunctionType: def task( task_callback: stb.type_hints.TaskCallbackType, task_abort_event: threading.Event, ) -> None: self.component_manager.do_assign( set(resources), task_callback=task_callback, task_abort_event=task_abort_event, ) return task .. warning:: The :py:func:`@validate_json_args ` decorator must always be applied first (innermost) to allow LRCs to catch validation errors and reject the task. Step 5: Optionally add an is-allowed method =========================================== The default method name of :meth:`!is__allowed` is automatically looked up on the device and called when the command is dequeued. If the method is omitted it will be assumed that the task is always allowed. An :meth:`!is__allowed` method must have a :obj:`!request_type` argument that defaults to :obj:`LRCReqType.ENQUEUE_REQ `, as Tango will automatically call it with no arguments when we enqueue the task. For example: .. code:: python class MyDevice(...): ... def is_VerySlow_allowed( self, request_type: stb.long_running_commands.LRCReqType | None = stb.long_running_commands.LRCReqType.ENQUEUE_REQ, ) -> bool: """Is 'VerySlow' command allowed. :return: True if the 'VerySlow' command can be executed """ ... .. note:: The is-allowed methods for all standard SKA commands are already implemented in the relevant base classes and are designed to work with both long running and fast commands. The :meth:`~ska_tango_base.long_running_commands.mixin.AbstractLRCMixin.submit_lrc_task` decorator will mark the :meth:`!execute_` methods to be treated as long running commands and this is used by the default is-allowed method implementation to decide when to check if execution is allowed. Step 6: Optionally implement the unhandled exception callback ============================================================= If your device inherits from :class:`~ska_tango_base.base.base_device.SKABaseDevice` or :class:`~ska_tango_base.long_running_commands.mixin.LRCMixin`, or component manager from :class:`~ska_tango_base.executor.executor_component_manager.TaskExecutorComponentManager`, you should implement the :meth:`!_on_unhandled_exception` method. It is called when the :class:`~ska_tango_base.executor.executor.TaskExecutor` catches any unhandled exceptions during execution of an LRC. It implies a bug in the device code, and the callback should be used to notify users thereof. Here is an example where the callback sets the device's state to ``FAULT``. .. code:: python class MyDevice/MyComponentManager: ... def _on_unhandled_exception(self, exception: Exception): self._update_component_state(fault=True)