Troubleshooting States and Modes
================================

For ``DishManager`` most commands will be rejected if the device is not in the correct ``dishMode``. When diagnosing state issues, the core rule is: *DishManager aggregates its state from its subservient devices.*

.. note:: Quick checks before diving into specific issues, verify the following:

    -   The device is in the expected state and mode
    -   All subdevices are reachable.
  
        -   This can be done by evaluating the connection state of each subdevice using a DeviceProxy to DishManager and checking the attributes: ``spfConnectionState``, ``spfrxConnectionState`` & ``dsConnectionState``.
  
    -   No errors are present in the device logs.

Scenario 1: Command Rejected 
-----------------------------

**Issue:** You attempt to run a command (e.g. ``Track``), with a ``dishMode`` at ``STANDBY-LP`` and receive a ``Command Rejected`` response.

**Root Cause:** You are trying to execute a command in a ``dishMode`` that is does not allow the particular command. For example, you cannot run a ``Track`` if the ``dishMode`` is in ``STANDBY-LP``, it must first be transitioned to ``dishmode`` ``OPERATE`` via the one of the ``ConfigureBand`` `commands <https://developer.skao.int/projects/ska-mid-dish-manager/en/latest/api/devices/dish-manager-device.html#commands>`_. 

For more information detailing the conditions under which each ``dishMode`` is reported, please refer to the Dish States and Modes ICD available `here <https://confluence.skatelescope.org/display/TDT/MID+Dish+States+and+Modes>`_. Additionally, visit the `DishManager command fanout page <https://confluence.skatelescope.org/pages/viewpage.action?pageId=188656205>`_ for a list of preconditions required for specific ``dishMode`` transitions.

**Resolution:**
Check the current dishmode using ``itango``.

.. code-block:: python

    import tango
    dish = tango.DeviceProxy("mid-dish/dish-manager/SKA001")

    # Check current mode and state
    dish.dishMode

    # Try transitioning to the modes for operation 
    dish.SetStandbyFPMode()

    # Init the track table
    dish.trackTableLoadMode = TrackTableLoadMode.NEW
    dish.programTrackTable = track_table
    # Now you should be able to execute the Track command
    dish.Track()



Scenario 2: DishManager Stuck in dishMode UNKNOWN 
-------------------------------------------------

**Issue:** When trying to issue a command that causes the ``dishMode`` to transition through a sequence of states and eventually reach the desired state, in some cases the ``dishMode`` reaches and gets stuck in ``UNKNOWN``.

**Root Cause:** This can occur when one of the subservient devices is not responding or is in an error state, resulting in the dish manager resolving the dish mode to ``UNKNOWN``.

**Resolution:** To recover from this state attempt to stow the dish using the command ``SetStowMode`` on the DishManager interface. Once the ``dishMode`` has successfully transitioned to ``STOW``, try transitioning the dish to low or full power mode using the command ``SetStandbyFPMode`` or ``SetStandbyLPMode`` to get the ``dishMode`` into a known state. Also refer to the `States and Modes <https://ska-telescope.gitlab.io/-/mid-dish/ska-dish-lmc/en/latest/user_guide/states_and_modes.html>`_

.. code-block:: python

    # Stow the dish to get it into a known state
    # Wait for mode STOW, to be reported on the dishMode attribute
    dish.Stow()
    # Check that Stow command was successful and dish is now in STOW mode
    dish.Status()
    # Verify that the dish is indeed in STOW mode
    dish.dishMode
    # Transition to low power mode
    dish.SetStandbyLPMode()
    # Verify the dish mode is now in a known state i.e STANDBY-LP
    dish.dishMode


.. tip:: Tracking LRC progress during execution
    
    Using a DeviceProxy to the DishManager device, the progress of the current command being executed can be checked by monitoring the `Status` attribute. 

    .. note:: If there are multiple commands in the queue and it is unclear which command was last invoked, we can check what the last command invoked was by reading the `lastcommandinvoked` attribute.