SKA LMC Queue Connector Tango Device

This project implements a Tango device intended to provide services to exchange data between various components internal and external to the Science Data Processor (SDP). Initially, this will essentially only be exchanging data between Tango devices external to the SDP and Kafka topics only available internally to SDP components, though it is reasonable to expect this may expand to other sources and sinks in future.

The LMC Queue Connector Device will sit in the Science Data Processor and interact with various other components both within and outside that boundary. These components will directly interact by reading from or subscribing to attributes on the Queue Connector device. They will also indirectly interact with the Data Exchange device by publishing data as a Tango attribute (if it’s a Tango device) or to a Kafka topic, which the Queue Connector device will subscribe to.

In the diagram below, we describe expected flow of data between each of these components, with the Queue Connector device essentially acting as a middle-man between Tango and Kafka.

One example workflow which this is expected to be used for is to calculate and publish pointing offsets to the telescope. Our Queue Connector Device will subscribe to the telescope sub-array device to obtain pointing data and write it to a Kafka topic, this will then be consumed by a processor in the realtime receive pipeline which will perform some calculations using the pointing data and the visibilities it is receiving. It will then write these offsets to another Kafka topic, which the the Queue Connector Device will be subscribed to, which is then exposed using a Tango attribute. The telescope sub-array will subscribe to this attribute to obtain the pointing offsets, completing the feedback loop.