Architecture

Exchange Model

The LMC Queue Connector Device processes data in the form of one or more configurable exchanges/streams that continuously process when the device is in the ON state.

The simplified class diagram for this can be expressed by the following:

_images/class-hierarchy.drawio.svg

Data Flow

An Exchange is fundamentally an object that asynchronously pulls data from one or more sources streams (DataSource), through a pipe operator (DataPipe), and then asynchronously pushes to a sink stream (DataSink). The asynchronous interface between sources and sinks allows for a single device to indefinately process multiple exchanges/streams, without threads being blocked waiting for I/O, and until instructed to stop.

Data into an exchange each use the same single dtype and input shape, and pipe operators may change the output data shape.

_images/data-flow.drawio.svg

Example

Two common configurations for exchanges are:

  • Publishing a Tango subscription to a Kafka topic

  • Publishing a Kafka topic as a Tango attribute

_images/expected-use-cases.drawio.svg

In this example there are 2 runtime implementations for each of DataSource and DataSink. Data is sourced from either another Tango device attribute or a Kafka topic. This data is then be propagated to either a local attribute on the Queue Connector device or to a Kafka topic. It is reasonable to expect the need for other implementations too (e.g. An in-memory Source and Sink for unit testing) and this is accomplished by implementing the DataSource and DataSink interfaces.