Architecture
Exchange Model
The LMC Queue Connector Device processes data in the form of one or more configurable exchanges/streams that continuously process when the device is in the ON
state.
The simplified class diagram for this can be expressed by the following:
Data Flow
An Exchange
is fundamentally an object that asynchronously pulls data from one or more sources streams (DataSource
), through a pipe operator (DataPipe
), and then asynchronously pushes to a sink stream (DataSink
). The asynchronous interface between sources and sinks allows for a single device to indefinately process multiple exchanges/streams, without threads being blocked waiting for I/O, and until instructed to stop.
Data into an exchange each use the same single dtype and input shape, and pipe operators may change the output data shape.
Example
Two common configurations for exchanges are:
Publishing a Tango subscription to a Kafka topic
Publishing a Kafka topic as a Tango attribute
In this example there are 2 runtime implementations for each of DataSource
and DataSink
. Data is sourced from either another Tango device attribute or a Kafka topic. This data is then be propagated to either a local attribute on the Queue Connector device or to a Kafka topic. It is reasonable to expect the need for other implementations too (e.g. An in-memory Source and Sink for unit testing) and this is accomplished by implementing the DataSource
and DataSink
interfaces.