============================================
How to extract data from a saved output file
============================================

The event monitor allows you to record the :func:`!QueryEventSystem` responses received from the monitor Tango devices to a file, but currently does not provide any mechanism to extract this data once it has been recorded.  This how-to will guide you through the process of extracting this data.

Iterating snapshots
-------------------

The file recorded with the ``--output`` argument passed to the ``ska-tango-event-monitor`` script records each snapshot as a separate JSON object on its own line.  These objects can be parsed by iterating over each line of the file and using `json.loads`:

.. code :: python

   import json

   snapshots = []
   with open("events.json") as f:
      for line in f:
         snapshots.append(json.loads(line))

.. tip :: If using the ``jq`` utility, this can be done with the ``--slurp`` option.

.. note :: If using compression, the appropriate compression library must be used to open the file.

Parsing snapshots
-----------------

Each snapshot has a ``"time"`` property that holds a string representation of the timestamp that the snapshot was taken at as well as a ``"replies"`` property holding a dictionary mapping admin device names to a response object.  Each response object holds either a ``"data"`` property, which holds the response to the :func:`!QueryEventSystem` command from that device server or a ``"error"`` property holding a string describing why the response was not collected.

- The ``"data"`` property for each device server follows the schema described in :ref:`response-schema` and can be parsed by :meth:`QueryEventSystem.from_json <ska_tango_event_monitor.QueryEventSystem.from_json>`.  For example:

.. code :: python

   import ska_tango_event_monitor as stem

   response = stem.QueryEventSystemResponse.from_json(snapshots[1]["replies"]["dserver/MyServer/1"]["data"])

Generating statistical summaries
--------------------------------

If performance monitoring was enabled then the response will have associated lists of :class:`~ska_tango_event_monitor.ServerPerfSample` and/or :class:`~ska_tango_event_monitor.ClientPerfSample` holding performance data.  These lists can be summarised using :meth:`ServerPerfSummary.from_samples <ska_tango_event_monitor.ServerPerfSummary.from_samples>` or :meth:`ClientPerfSummary.from_samples <ska_tango_event_monitor.ClientPerfSummary.from_samples>`.  For example:

.. code :: python

   stats = stem.ServerPerfSummary.from_samples(response.server.perf)