Observed CSP.LMC Subarray Behaviour =================================== This page describes the client-visible behaviour of the CSP.LMC Subarray ``obsState`` as produced by the Observation Supervisor. The page does not describe the internal implementation of the supervisor. Instead, it explains which CSP.LMC Subarray ``obsState`` a client can expect to see when subsystem state changes are observed during command execution, scan supervision, or normal monitoring. The CSP.LMC Subarray ``obsState`` must not be interpreted as a simple aggregation of the subsystem ``obsState`` values. It is the supervised operational state of the CSP.LMC Subarray, derived from the observed subsystem states and from the current CSP operational context. Client-visible Principle ------------------------ A client observes the CSP.LMC Subarray ``obsState`` as the authoritative state of the CSP.LMC Subarray. The Observation Supervisor ensures that the published CSP.LMC Subarray ``obsState`` is consistent with the current operational context. In particular, the same subsystem state may lead to different CSP.LMC Subarray states depending on whether the subsystem transition is: * expected as part of a running command; * expected as part of a recovery sequence; * expected as part of an active scan lifecycle; * unexpected because the CSP.LMC Subarray is otherwise stable and no command is running. For example, a CBF transition to ``EMPTY`` is expected during ``ReleaseResources``. The same CBF transition is unexpected if the CSP.LMC Subarray is stable in ``READY`` and no command is active. While a CSP command is running, the CSP.LMC Subarray obsState published to clients is driven by the command semantics. This means that subsystem state changes observed during the command are treated as part of the command execution, not as spontaneous state changes. For example, during Configure, one subsystem may reach READY before the others. In this case, the CSP.LMC Subarray does not necessarily publish READY immediately. It waits until the observed subsystem states are consistent with the expected result of the Configure command. Similarly, during ReleaseResources, CBF may report EMPTY. This is not treated as an error by itself, because EMPTY is an expected state during a resource release. From a client point of view, this means that intermediate subsystem events may not be reflected immediately as CSP.LMC Subarray obsState transitions. The published CSP.LMC Subarray obsState represents the supervised result of the command execution, not the latest individual subsystem event. Behaviour after Command Completion ---------------------------------- A command may complete before all subsystem state events have been received. After command completion, the CSP.LMC Subarray evaluates the latest observed subsystem states and derives the corresponding aggregate `obsState`. If the aggregated state is compatible with the state expected for the completed command, the CSP.LMC Subarray may publish the corresponding transition even if not all subsystems that successfully executed the command have already reported the expected final state. For example, after `Configure` completes, the CSP.LMC Subarray may publish `READY` when the aggregated subsystem state is consistent with a successful configuration. This does not necessarily imply that every subsystem involved in the command has already published `READY`. From a client point of view, the published CSP.LMC Subarray `obsState` represents the supervised aggregate state currently considered consistent with the completed command, rather than an explicit confirmation that every successful subsystem has individually reached the expected final state. If a command fails, the CSP.LMC Subarray behaviour reflects the command failure, not only the aggregate state currently reported by the subsystems. For example, if `Restart` is invoked while the CSP.LMC Subarray is `ABORTED` and the command fails, the CSP.LMC Subarray reports `FAULT`. This makes the failed recovery attempt visible to the client, even if the subsystems are still observed in their previous states. For example: .. code-block:: text CSP.LMC Subarray obsState = ABORTED Restart command is invoked Restart command returns FAILED CSP.LMC Subarray obsState = FAULT A failed ``Restart`` is a failed recovery attempt. Therefore, the client sees ``FAULT`` rather than a silent return to the previous state. Behaviour during Scan Supervision --------------------------------- During scan supervision, clients should expect the CSP.LMC Subarray ``obsState`` to represent the supervised scan lifecycle. The ``Scan`` command may complete before all relevant subsystem events have been received. For this reason, the CSP.LMC Subarray may still apply scan-specific semantics after the ``Scan`` command itself has completed. Typical client-visible behaviour is: .. list-table:: :header-rows: 1 :widths: 30 35 35 * - Client-visible situation - Subsystem observation - Expected CSP.LMC Subarray behaviour * - ``Scan`` completed, but subsystem events are not yet converged - Relevant subsystems are still ``READY``. - CSP.LMC Subarray remains ``READY``. * - Scan lifecycle is active - Relevant successful subsystems report ``SCANNING``. - CSP.LMC Subarray reports ``SCANNING``. * - Scan is ending - Relevant successful subsystems return to ``READY``. - CSP.LMC Subarray reports ``READY``. * - Core subsystem is lost during scan - CBF reports ``EMPTY``. - CSP.LMC Subarray reports ``FAULT`` or applies the configured scan failure policy. From a client point of view, the CSP.LMC Subarray reports ``SCANNING`` only when the scan is actually consistent with the observed state of the relevant subsystems. Behaviour during Normal Monitoring ---------------------------------- The CSP.LMC Subarray is in normal monitoring when no command is running and no scan lifecycle is being supervised. In this condition, subsystem transitions are interpreted as spontaneous events. If a required subsystem unexpectedly changes state in a way that is not compatible with the current CSP.LMC Subarray state, the CSP.LMC Subarray exposes the inconsistency to the client. The most important example is an unexpected CBF restart. Unexpected CBF ``EMPTY`` ------------------------ CBF is a core subsystem for the CSP.LMC Subarray. If CBF unexpectedly reports ``EMPTY`` while the CSP.LMC Subarray is stable in another observation state, the client must not see a false clean transition to ``EMPTY``. The CSP.LMC Subarray behaviour depends on the client-visible context. .. list-table:: :header-rows: 1 :widths: 30 35 35 * - Client-visible context - CBF observation - CSP.LMC Subarray behaviour * - ``ReleaseResources`` is running - CBF reports ``EMPTY``. - CBF ``EMPTY`` is expected. The CSP.LMC Subarray follows the command result. * - ``Restart`` or ``ObsReset`` is running - CBF reports ``EMPTY``. - CBF ``EMPTY`` may be part of the recovery sequence. * - CSP.LMC Subarray is stable in ``READY`` - CBF reports ``EMPTY`` without a CSP command. - CSP.LMC Subarray reports ``FAULT``. * - CSP.LMC Subarray is stable in ``IDLE`` - CBF reports ``EMPTY`` without a CSP command. - CSP.LMC Subarray reports ``FAULT``. * - CSP.LMC Subarray is ``SCANNING`` - CBF reports ``EMPTY``. - CSP.LMC Subarray reports ``FAULT`` or applies the configured scan failure policy. * - CSP.LMC Subarray is already ``ABORTED`` - CBF reports ``EMPTY``. - CSP.LMC Subarray remains ``ABORTED``. The ``ABORTED`` case is different because the CSP.LMC Subarray is already in a recovery state. The client should continue to see ``ABORTED`` so that the operator can use the expected recovery path, such as ``Restart``. In contrast, if the CSP.LMC Subarray is stable in ``READY`` or ``IDLE``, an unexpected CBF transition to ``EMPTY`` means that CBF has restarted or lost its resources outside CSP control. In that case, the client sees ``FAULT`` because the CSP.LMC Subarray state is no longer coherent with the state of a required subsystem. Example: CBF restarts while CSP is ``READY`` -------------------------------------------- Initial client-visible state: .. code-block:: text CSP.LMC Subarray obsState = READY Subsystem event: .. code-block:: text CBF obsState = EMPTY No CSP command is running. Expected client-visible result: .. code-block:: text CSP.LMC Subarray obsState = FAULT Rationale: The CSP.LMC Subarray was configured and ready. CBF restarted or released its resources outside CSP control. The CSP.LMC Subarray cannot honestly report ``READY``, because a required subsystem is no longer in the expected state. It also must not report ``EMPTY``, because the CSP.LMC Subarray was not released by a CSP command. Reporting ``FAULT`` makes the inconsistency visible to the client and allows the operator to recover the system through the expected recovery path. Example: CBF reports ``EMPTY`` during ``ReleaseResources`` ---------------------------------------------------------- Initial client-visible state: .. code-block:: text CSP.LMC Subarray obsState = IDLE Command: .. code-block:: text ReleaseResources is running Subsystem event: .. code-block:: text CBF obsState = EMPTY Expected client-visible result: .. code-block:: text CBF EMPTY is treated as expected command progress. The CSP.LMC Subarray follows the ReleaseResources command result. Rationale: In this case, CBF ``EMPTY`` is not an inconsistency. It is the expected result of a CSP-controlled resource release. Example: CBF restarts while CSP is ``ABORTED`` ---------------------------------------------- Initial client-visible state: .. code-block:: text CSP.LMC Subarray obsState = ABORTED Subsystem event: .. code-block:: text CBF obsState = EMPTY No CSP command is running. Expected client-visible result: .. code-block:: text CSP.LMC Subarray obsState remains ABORTED Rationale: The CSP.LMC Subarray is already in a recovery state. The client should continue to see ``ABORTED`` because the expected operator action is still to recover the system, for example by invoking ``Restart``. The CBF transition to ``EMPTY`` does not need to promote the CSP.LMC Subarray to ``FAULT`` in this case. Client-visible Contract ----------------------- Clients can rely on the following behaviour: * the CSP.LMC Subarray ``obsState`` is context-aware and is not a plain aggregation of subsystem states; * subsystem transitions caused by active commands are interpreted as command progress; * scan-related subsystem transitions are interpreted according to the active scan lifecycle; * unexpected subsystem transitions during normal monitoring are exposed as CSP inconsistencies; * an unexpected CBF transition to ``EMPTY`` while CSP is stable in ``READY`` or ``IDLE`` is reported as ``FAULT``; * an unexpected CBF transition to ``EMPTY`` while CSP is already ``ABORTED`` preserves the ``ABORTED`` recovery path; * failed recovery commands, such as a failed ``Restart``, are visible to the client as ``FAULT``.