Remoting

This section gives a summary of the main principles involved in the realisation of remote connectors (see ska_ser_skallop.connectors.remoting).

Background

The skallop package connects to the System Under Test (SUT) via abstract connectors that can be realised in a number of different ways. However, there are primarily two kind of environments in which these connectors are realised:

1. Build In Test environment (BIT)

2. Build Out Test environment

In a BIT environment the connectors are realised using the tango framework, providing client connections through the Tango DeviceProxy communicating with a K8 tango database service. This is the same method with which a large part of control and monitoring within the SUT takes place and thus gives an environment in which the test itself is “build in” to the SUT.

The disadvantage is that developing of tests requires the full deployed SUT infrastructure for the test itself, making developing from a remote developer’s machine cumbersome. The build out test environment aims to realise the connectors by making use of externally provided API’s to the SUT as http REST and websocket interfaces, ensuring a developer can run the test from his own machine communicating to the SUT over http. However to make use of build out test implementation the SUT must be provided with an external public API. This is implemented by the Tango GQL service, primarily used by the Taranta application.

Thus skallop provides the user with a bridge, nl TangoBridge, acting as the client for communicating to this Tango GQL Service. The next sections will explain how that has been designed.

Domain Model

The diagram below explains the concept by means of a basic UML domain model (only key relationships are shown).

RemoteDevicesQuery and RemoteDeviceProxy implements AbstractDeviceProxy and AbstractDevicesQuery respectively using the TangoBridge singleton. (Note the Producer class is also realised by RemoteDeviceProxy since Producer is defined as being a “superclass” of AbstractDevicesQuery)

TangoBridge is thus responsible for providing a facade to these components giving them access to two main operations:

1. Call graphql: Calls a rest based graphql query to the tangogql service

2. Add/Remove subscriptions: Create/Remove subscriptions on the graphql service for tango devices

Note

Note the TangoBridge is a singleton, meaning that at any moment in time, a number of clients may be calling on the same instance.

As the single instance for all connections to the tangogql service, the TangoBridge has the additional “managerial” responsibility for:

1. overseeing the health of current connections;

2. ensuring authentication to get access to the given tango gql service.

It will also use a single websocket connection for all subscribing connections to the external service ensuring scalability in number of devices needing to create subscriptions.

Subscriptions between a device proxy subscriber and the tango gql service are mediated via the DeviceAttributeSubscriber providing the WSController with subscribers (see Subscriber) to websocket events, filtered and parsed into tango device attribute events and finally resulting in calls to given device proxy callbacks (see DeviceSubscriptionCallback). The TangoBridge, being a singleton, also “multiplexes” subscriptions to the service if they are for the same device and attribute as a single subscription in order to reduce traffic.

The monitoring parts of the websocket and rest interfaces are done via controller loops periodically calling or listening for key events within the RestController and WSController objects. These “daemon like” loops are run as asyncio Tasks (see asyncio.Task) on a separate asyncio thread, owned and managed by the Controller object. The Controller owns a main daemon thread and is responsible for dispatching and tearing down asyncio tasks (as either blocked or concurrent threads) on this thread. The RestController and WSController can therefore make asyncio calls (as co-routines) to the tangogql service on an event loop separate from the main thread.

Authentication

The domain model for authentication related to the rest connection (not shown in main model) is detailed in diagram below:

During initialisation, the RestController makes use of the TangoBridgeFactory to obtain an AuthenticatedUser data object. The TBridgeFactory in turn uses the Authenticator to obtain the data by logging in to the tango gql service using a given set of Environment credentials (usually obtained from the host environment). If authentication was successful, the RestController is provided with credentials (as Header cookies) for making secure http calls to the tango gql service.

Warning

The current design does not use secure messages on the websocket connection. However, access to the websocket can only be done if a secure VPN network to the IP address exists. Therefore remoting currently has the additional constraint of requiring users to have an appropriate VPN connection.

Tango Bridge Monitoring and Control Design

Concurrency

Tangobridge perform all its monitoring anc control tasks in a separate thread (named asyncio) managed by Controller.

However these tasks are single threaded within the asyncio thread as asyncio co-routines.

As depicted in the diagram below: A task (python co-routine) is loaded/dispatched for execution onto the asyncio thread’s event loop. When the thread is not running it will activate that task and run it until it either get’s paused (await) or finishes. It will then take the next task and perform the same sequence that task.

Note, a task can also create other concurrent tasks from within itself. Thus the asyncio thread can have two ways of getting tasks:

1. From the main thread (returning a Future)

2. From the asyncio thread as a result of a task dispatching a co-routine on the event loop (returning a asyncio.Task)

Thread Life Cycle

During Initialisation of the tangobridge, the factory call to get a controller results in the asyncio thread being created. This thread is set as a daemon and will get destroyed after the main thread stops. Once the controller is initialised, it is used by the WSController and RestController to create a monitoring loop as a asyncio tasks. The Controller also runs its own monitoring loop to periodically poll the status of running tasks.

At the end of the program the WSController and RestController signals their controlling loops to stop, followed by the Controller signalling it’s loop to stop. Thereafter the Controller will cancel any tasks still pending for executing before it’s asyncio daemon thread get’s destroyed.

WSController and RestController use their main monitoring loops to dispatch additional co routine Tasks according to their needs and responsibilities:

The following list describes the main tasks launched by WSController:

1. Receive websocket messages

2. Sends new messages placed in incoming queue

3. Monitor websocket health

The RestController does not dispatch any additional tasks within it’s loop but simply polls the tango gql service periodically to see if it is still healthy.