How to integrate ska-tango-event-monitor with your CI pipelines
===============================================================

Currently, this utility is manually applied within shell sessions on pods deployed in both local and ITF clusters.

Multiple Tango device servers can be monitored at once and it is recommend to monitor them all with a single monitoring process so that they are sampled at the same time.  This makes it easier to correlate the event system query data between devices.

Each device must be running against cppTango/pytango 10.1.0 or later.

To streamline usage, we propose adding a dedicated job to the test stage. This job will:

* Stream summarized event processing information during test execution.
* Capture and save events transmitted over the ZMQ wire to a file called `zmq-events.json`

This file will contain detailed event data, including device (server) publications, client subscriptions to attributes, and their respective callback counts.

To further simplify management, a make target will be integrated into the ska-cicd-makefile. In the interim, the pipeline structure from `this example pipeline <https://gitlab.com/ska-telescope/ska-mid-dish-manager/-/pipelines/1698758127>`_ can be replicated (for integration environment, wait for make target).

Steps to setup
--------------

- **Update or Add a Dockerfile to install ska-tango-event-monitor**:

.. code :: Dockerfile

   # ./build-with-custom-whl/Dockerfile
   RUN pip install ska-tango-event-monitor --extra-index-url https://artefact.skao.int/repository/pypi-internal/simple

- **Point your docker build command to the new dockerfile**

.. code :: Makefile

   # ./Makefile
   OCI_IMAGE_FILE_PATH = build-with-custom-whl/Dockerfile

- **Add jobs to stream and store the events**

.. code :: yaml

  # .gitlab-ci.yml
  stream-processed-zmq-events:
    tags:
      - ${SKA_K8S_RUNNER}
    variables:
      KUBE_NAMESPACE: 'ci-$CI_PROJECT_NAME-$CI_COMMIT_SHORT_SHA'
      TARGET_POD_NAME: <the-pod-to-run-the-sampling-script>
      DEVICE_NAMES: "<foo/bar/1> <foo/bar/2> ..."
    allow_failure: true
    when: always
    stage: test
    script:
      - git clone https://gitlab.com/ska-telescope/sdi/ska-cicd-makefile.git
      - cd ska-cicd-makefile
      - KUBE_APP=<k8s-app-label-name> make k8s-wait
      - echo "Starting ZMQ event monitoring on $DEVICE_NAME in namespace $KUBE_NAMESPACE"
      - kubectl exec -i $TARGET_POD_NAME -n $KUBE_NAMESPACE -- sudo touch zmq-events.json.xz
      - kubectl exec -i $TARGET_POD_NAME -n $KUBE_NAMESPACE -- sudo chown tango zmq-events.json.xz
      - kubectl exec -i $TARGET_POD_NAME -n $KUBE_NAMESPACE -- /app/bin/ska-tango-event-monitor $DEVICE_NAMES --monitor-perf --append --output zmq-events.json.xz

  stop-streaming-and-store-zmq-events:
    tags:
      - ${SKA_K8S_RUNNER}
    variables:
      KUBE_NAMESPACE: 'ci-$CI_PROJECT_NAME-$CI_COMMIT_SHORT_SHA'
      TARGET_POD_NAME: <the-pod-to-run-the-sampling-script>
    stage: test
    when: always
    allow_failure: true
    script:
      - echo "Test run has completed, collecting events recorded"
      - mkdir -p build
      - kubectl exec -i $TARGET_POD_NAME -n $KUBE_NAMESPACE -- cat zmq-events.json.xz >> build/zmq-events.json.xz
      - kubectl exec -i $TARGET_POD_NAME -n $KUBE_NAMESPACE -- pkill -f "/app/bin/ska-tango-event-monitor"
    needs:
      - k8s-test-runner
    artifacts:
      name: "$CI_JOB_NAME-$CI_JOB_ID-recorded-events"
      paths:
        - build/