Device: DishBaseDevice

A Tango Device Server (TDS) base class for DishLMC devices.

Copyright: 2019 Simone Riggi (INAF) License: GPL-3

Description

It is convenient to group common functionalities needed by all DSH.LMC Tango devices in a unique mother device. We named it DishBaseDevice. The functionalities that can be used in derived Tango devices are mostly provided by utility classes in src/ska/utils:

  • Standardized Logging

  • Command allowed rules

  • Proxy helper utilities

  • Task management

  • Event handling

We describe each one with more detail in the following sections.

Standardized Logging

This includes:

  • Logging to a syslog target using boost syslog

  • Logging patterns defined in the SKA Logging Guidelines

Logging macros

Logging macros are provided by the Logger helper class (add #include <Logger.h> in your header list) to send a log message to enabled log targets (central, element, storage targets) . The logging macros internally perform logging to Tango device targets and also logging to syslog as described in SKA Logging Guidelines.

Here are the available macros to be used in different situations:

  1. Inside Tango devices: _FATAL_LOG, _ERROR_LOG, _WARN_LOG, _INFO_LOG, _DEBUG_LOG

    Example: _INFO_LOG(“This is an info log message with data “<<data<<”…”);

  2. Outside Tango devices: prepend a double underscore in log macro and pass a device instance (Tango::DeviceImpl*) to log in the name of that device.

    Example: __ERROR_LOG(dev,”This is an info log message with data “<<data<<”…”);

The above macro automatically log towards the attached and enabled log targets (including syslog) at configured levels. The log message sent to Tango log targets (console/file/device) is made up of a predefined part (automatically set by the Tango core) plus a message formatted as follows:

[CLASS NAME]::[FCN NAME]() - Log message

Example: 1482360581 [140324827555584] INFO mid_dsh_0001/elt/master EventCallBack::process_event() - Event received

NB: Need to fix log message source information for templated methods.

The message format for syslog target is instead configurable from the rsyslog configuration files. See examples reported in the SKALoggingGuidelines.

Logging Configuration

Three predefined logging targets can be attached to each device in conformance with SKA Logging Guidelines:

  1. Element log target: a Tango Log Consumer device present in the Tango Dish LMC facility (dubbed ElementLogger)

  2. Central log target: a Tango Log Consumer device present in the Tango TM-Dish facility (dubbed CentralLogger)

  3. Storage log target: a rsyslog server running in the Dish LMC system.

Default logging levels and target names can be configured from a set of device properties:

  • Central log target

    • CentralLoggerEnabledDefault: To enable/disable logging towards Central log target device (aka CentralLogger)

    • LoggingTargetCentralDefault: Pre-configured Central logging target, i.e. CentralLogger device server name

    • LoggingLevelCentralDefault: Default logging level to Central logging target (0=OFF, 1=FATAL, 2=ERROR, 3=WARNING, 4=INFO, 5=DEBUG). The default according to SKA Logging Guidelines is 3=WARNING

  • Element log target

    • ElementLoggerEnabledDefault: To enable/disable logging towards Element log target device (aka ElementLogger)

    • LoggingTargetElementDefault: Pre-configured Element logging target, i.e. ElementLogger device server name

    • LoggingLevelElementDefault: Default logging level to Element logging target (0=OFF, 1=FATAL, 2=ERROR, 3=WARNING, 4=INFO, 5=DEBUG). The default according to SKA Logging Guidelines is 4=INFO.

  • Storage log target

    • StorageLoggerEnabledDefault: To enable/disable logging towards rsyslog server

    • LoggingTargetStorageDefault: Pre-configured Storage logging target, e.g. hostname of rsyslog server (e.g. localhost)

    • LoggingLevelStorageDefault: Default logging level to Element logging target (0=OFF, 1=FATAL, 2=ERROR, 3=WARNING, 4=INFO, 5=DEBUG). The default according to SKA Logging Guidelines is 4=INFO.

    • SyslogFacility: Syslog facility flag, e.g. set to local6 by default. See device property doc for additional info on alternative values.

Logging targets can be modified at runtime using the following memorized writable attributes (e.g. targets are persisted after device restart):

  • loggingTargetCentral: Current Central log target

  • loggingTargetElement: Current Element log target

  • loggingTargetStorage: Current Storage log target

Logging levels per target can be modified at runtime using the following memorized writable attributes (e.g. levels are persisted after device restart):

  • loggingLevelCentral: Current log level for Central log target

  • loggingLevelElement: Current log level for Element log target

  • loggingLevelStorage: Current log level for Storage log target

All the listed properties and attributes are defined in the base device (DishBaseDevice) and are automatically added in your device if you inherit from it. You must only take care of registering them in the TangoDB, e.g. manually using Jive or programmatically using scripts.

Command allowed rules

Command state machine rules can be defined in Tango only using the device State attribute. For this you just need to add states to the device and configure the State Machine in Pogo. Pogo automatically generate for each command XXX a is_XXX_allowed(TANGO_UNUSED(const CORBA::Any &any)) method in your XXXStateMachine.cc file with State rule checked. We would like command rules defined with other scalar attribute types (particularly DevEnum and DevBoolean) defined in the device. To this aim we provided this functionality in the utility classes and defined a virtual method named int DishBaseDevice::InitCmdAttrStateMachine() that is automatically called inside the add_dynamic_attributes() method of base device. In derived devices you may override this method to add the command rules as shown below:

int MyDerivedDevice::InitCmdAttrStateMachine()
{
  //The rule requires that the ConfigureBand command can be inkoved only when rxOperatingMode attribute is in DATA-CAPTURE/MAINTENANCE/STANDBY-FP
  _REGISTER_CMD_RULE(
     std::string,
     "ConfigureBand",
     "rxOperatingMode",
     ("DATA-CAPTURE","MAINTENANCE","STANDBY-FP")
  );

  //The rule requires that the ConfigureBand can be inkoved only when b1CapabilityState attribute is in STANDBY/OPERATE
  _REGISTER_CMD_RULE(
    std::string,
    "ConfigureBand",
    "b1CapabilityState",
    ("STANDBY","OPERATE")
  );

  //...
}

void MyDerivedDevice::add_dynamic_attributes()
{
  //Call base method to add dynamic attributes
  //NB: This is calling InitCmdAttrStateMachine() and other utility functions
  DishBaseDevice::add_dynamic_attributes();

  //...
}

Alternatively you can call this method inside the init_device() of derived devices. Once you have registered the rules you can use the method Utils_ns::TangoUtils::IsCmdAllowed(Tango::DeviceImpl* dev,std::string cmd_name,bool check_cmd) method in your command or inside the is_XXX_allowed method to check if command is allowed or not. For example:

bool MyDerivedDevice::is_MyCmd_allowed(TANGO_UNUSED(const CORBA::Any &any))
{
  bool check_cmd= true;
        try {
          Utils_ns::TangoUtils::IsCmdAllowed(this,__FUNCTION__,check_cmd);
        }
        catch(Tango::DevFailed& e){
                Tango::Except::print_exception(e);
                throw;
        }

  //...
}

Proxy helper utilities

A number of proxy utilities are provided by the TangoProxyManager class, listed in the following subsections.

Proxy registration

To add a proxy in your device server you need to register it in the TangoProxyManager class. You have different possibilities:

  • Manual registration: Override the int RegisterProxies() virtual method provided in the DishBaseDevice in your derived devices and use the _REGISTER_PROXY(<proxy_name>) method. The example below registers a proxy (mid_dsh_0001/rx/controller) in a device MyDevice:

    int MyDevice::RegisterProxies()
{
  _REGISTER_PROXY("tango:://localhost:10000/mid_dsh_0001/rx/controller");
  //...

  return 0;
}

  • Registration from properties: You can also specify proxies to be registered in the ProxyDevices string list property. Proxies listed in the property will be automatically registered at device startup right before the RegisterProxies method seen before is executed.

NB: You must always use FQDN to specify proxy names in all the discussed cases.

You can unregister a proxy from the device using the corresponding device method _UNREGISTER_PROXY(proxy_name). Both register and unregister methods use internally the TangoProxyManager class methods: Register(proxy_name,monitoring_timeout)/UnregisterProxy(proxy_name). Once you have registered a proxy you can retrieve it from the collection, using the TangoProxyManager method Tango::DeviceProxy* FindProxy(proxy_name), and use all the standard Tango client API methods on it.

Proxy registration can fail if proxy is offline at the moment of registration or if proxy is not existing (e.g. you have made a mistake in the proxy name) or not exported in DB. In the first case no error is thrown, proxy is inserted in the list and initialization is periodically performed again by the polled command DishBaseDevice::InitializeDevProxies(). Note that probably this step is not needed because the Tango core automatically tries to reconnect to a proxy device. However this is not the case for event subscriptions. Event subscriptions fail if proxy is offline and are never performed again. For this reason this command also periodically retries to subscribe to proxy events. If event subscription succeeded the subscriptions is not repeated and the Tango core automatically monitors the event channel via heartbeats.

Proxy monitoring

A polled command DishBaseDevice::MonitorProxies() is defined to periodically ping each registered proxy and store proxy monitoring information (e.g. run status, device State & Status, ping instantaneous and averaged time, time elapsed in current run status). When proxy is responding the run status is set to ONLINE. When proxy is not responding the run status is set to SOFT-OFFLINE. If downtime persists for more than a configurable timeout interval proxy the run status is switched to HARD-OFFLINE. A virtual method DishBaseDevice::DoPostProxyMonitoringActions()` is automatically called inside MonitorProxies() after proxy monitoring. You can override this method in derived device and implement a series of actions to be performed after proxy monitoring. For example you can push events on proxy downtime, catch them on other devices and react as desired, e.g. setting given attribute quality to Tango::ATTR_INVALID.To monitor a proxy and get its run status inside the device you can use the TangoProxyManager method MonitorProxy(proxy_name,&proxy_info).

Proxy event subscription

Similarly to what discussed for proxy registration you can register event subscriptions in your device in different ways:

  • Manual registration: Override the int RegisterEventSubscriptions() virtual method provided in the DishBaseDevice and use the _REGISTER_EVT_SUBSCRIPTIONS(<proxy_name>,<attr_name>,<evt_types>) method. The example below registers in a device MyDevice subscriptions to periodic/change/user events for attribute rxOperatingMode in proxy (mid_dsh_0001/rx/controller):

    int MyDevice::RegisterEventSubscriptions()
{
  std::string proxy_name("tango:://localhost:10000/mid_dsh_0001/rx/controller");
  _REGISTER_EVT_SUBSCRIPTIONS(
    proxy_name,
    "rxOperatingMode",
    {Tango::CHANGE_EVENT,Tango::PERIODIC_EVENT,Tango::USER_EVENT}
  );

  //...

  return 0;
}

  • Registration from properties: You can also specify proxies to be registered in the EventSubscriptions string list property. All events listed in the property will be automatically registered at device startup right before the RegisterEventSubscriptions method seen before is executed. You must follow this format to define the list:

    [0]= Full attribute name 1 (e.g. "tango:://localhost:10000/mid_dsh_0001/rx/controller/rxOperatingMode")
[1]= Event type 1 (e.g. "change", "periodic", "user", ...)
...
[N-1]= Full attribute name N
[N]= Event type N

NB: You must always use FQDN to specify full event names (proxy+attr name) in all the discussed cases.

Event subscription info are stored in the EvtSubscriptionData class and can be managed via the TangoProxyManager. If event subscription is successful (e.g. proxy is online) the registered events are received and processed in a dedicated thread by the EventCallBack class. The functionalities implemented for event processing are described in the Event handling section. If event subscription fails (e.g. proxy offline) the device will retry it on a periodical basis in the polled command InitializeDevProxies.

Task management

According to requirements LMC should support execution of commands (or of sequence of commands) at deferred times (e.g. with a specified time delay wrt to invocation time) or at given timestamp with timeout specification with dependencies among sub-tasks in the sequence. This functionality is currently not provided by Tango which only supports command execution as soon as they are called. Furthermore there is no builtin concepts allowing to implement sequence of operations or a mechanism to prioritize tasks. The yat4tango library partially provides these functionalities: the user has to implement a yat4tango::DeviceTask per each task and stimulate task activation with a given message posted from device commands. Periodic and timeout actions are also supported. To better fulfill our use cases we implemented a task queue in the DishBaseDevice_ns::TaskThread using the task utility classes. Tasks in a sequence are pushed in the task queue according to their priority, activation timestamp and mutual dependencies and inserted also in a history list for monitoring purposes. A dedicated thread (the TaskThread class) pops the top task from the queue, encode its data (arguments, timestamp, etc) in a JSON string and execute the task command with json string as argument. Popped tasks are consumed and therefore removed from the queue. The queue can be filled up to a maximum and configurable number of tasks. When quota is reached no tasks can be added (e.g. an exception is thrown). Tasks (both queued, running and completed) are maintained in a monitoring list up to a maximum configurable number. Completed tasks, older than a configurable time interval, are periodically removed from this historical list. Commands are provided in the DishBaseDevice to get the number of sequence tasks in the queue by state, query sequence run status, get task info, etc.

The following example illustrate how to define and run a task sequence (e.g. dish band configuration) in a device:

//====================================
//==   DEFINE SUB TASKS & SEQUENCE
//====================================
bool useUTC= false;
int priority= Utils_ns::Task::eMEDIUM_PRIORITY;
int configureTaskTimeout= 60;//in seconds
int rxBandId= ...
int dsBandId= ...
int spfBandId= ...

//***********************
//**   Rx SUB TASKS    **
//******************* ***
//- Task 1: Rx ConfigureBand
std::string rx_task1_name= "ConfigureRxBandTask";
CORBA::Any rx_task1_argin;
rx_task1_argin <<= rxBandId;

Utils_ns::TaskPtr rx_task1= std::make_shared<Utils_ns::Task>(this,rx_task1_name,priority,useUTC);
rx_task1->SetMaxDuration(configureTaskTimeout*1000.);//Set task timeout
rx_task1->SetTaskArgin(rx_task1_argin,Tango::DEV_SHORT,Tango::SCALAR);//set task argin

//- Task 2: Rx Synchronise band (depend on Task1)
std::string rx_task2_name= "SynchroniseRxBandTask";
Utils_ns::TaskPtr rx_task2= std::make_shared<Utils_ns::Task>(this,rx_task2_name,priority,useUTC);
rx_task2->SetMaxDuration(defaultTaskTimeout*1000.);//Set task timeout

//***********************
//**   DS SUB TASKS    **
//***********************
//- Task 1 (long running cmd): DS move indexer
std::string ds_task1_name= "SetIndexerPositionTask";
std::string ds_task1_progress_attr_name= "dsIndexingProgress";
CORBA::Any ds_task1_argin;
ds_task1_argin <<= dsBandId;

Utils_ns::TaskPtr ds_task1= std::make_shared<Utils_ns::Task>(this,ds_task1_name,priority,useUTC);
ds_task1->SetMaxDuration(configureTaskTimeout*1000.);//Set task timeout
ds_task1->SetTaskArgin(ds_task1_argin,Tango::DEV_SHORT,Tango::SCALAR);//set task argin
ds_task1->SetAsynchTask(ds_task1_progress_attr_name);//Set task progress attr name

//************************
//**   SPF SUB TASKS    **
//************************
//- Task 1: SPF set band in focus (depend on DS task)
std::string spf_task1_name= "SetSPFBandInFocusTask";
CORBA::Any spf_task1_argin;
spf_task1_argin <<= spfBandId;

Utils_ns::TaskPtr spf_task1= std::make_shared<Utils_ns::Task>(this,spf_task1_name,priority,useUTC);
spf_task1->SetMaxDuration(defaultTaskTimeout*1000.);//Set task timeout
spf_task1->SetTaskArgin(spf_task1_argin,Tango::DEV_SHORT,Tango::SCALAR);//set task argin

//************************
//**   TASK SEQUENCE    **
//************************
//- Create task sequence
std::string taskSequence_name= "ConfigureBand";
std::string taskSequence_progress_attr_name= "configureBandProgress";
Utils_ns::TaskSequencePtr taskSequence= std::make_shared<Utils_ns::TaskSequence>(this,taskSequence_name,useUTC);
taskSequence->SetDevStatusAttr(taskSequence_progress_attr_name);//Set device attr containing task sequence progress status

//- Add tasks to sequence
taskSequence->AddTask(rx_task1);
taskSequence->AddTask(rx_task2);
taskSequence->AddTask(ds_task1);
taskSequence->AddTask(spf_task1);

//- Set task dependencies
std::vector< std::vector<size_t> > task_dependency_matrix
{
  {},   //Rx task 1  (not depending on other tasks)
  {0},  //Rx task 2  (depending on Rx task 1)
  {},   //DS task 1  (not depending on other tasks
  {2}   //SPF task 3 (depending on DS task 1)
};

taskSequence->SetTaskDependency(task_dependency_matrix);

//==============================
//==      SCHEDULE TASKS
//==============================
//- Add task to queue
bool addToQueue= true;
if(m_taskManager->AddTaskSequence(taskSequence,addToQueue)<0)
{
  std::stringstream ss;
  ss<<"Failed to add task sequence "<<taskSequence_name<<" to queue!";
  _WARN_LOG(ss.str());
  _THROW_TANGO_EXCEPTION("CONFIGURE_BAND_FAILED",ss.str());
}
_INFO_LOG("Added sequence "<<taskSequence_name<<" to task queue...");

//Set task sequence status to IDLE
//...

The above code defines a task sequence (named ConfigureBand) made by four interdependent tasks to be executed. Second task is depending on the first to start and last one is depending on third. Task names (e.g. ConfigureRxBandTask, etc) must correspond to device commands which accepts a DevString argin representing json encoded task data (see example below). Task 1, 2 and 4 completes as soon as command response is provided, e.g. response is used to set task status. Task 3 (SetIndexerPositionTask) is a long-running task. It provides a prompt response to notify that requested action started on the system. This is used to set task status initially but task status has to be monitored over time using a progress attribute (in this case dsIndexingProgress). Task sequence status is computed periodically from sub task run status. Each task can be also given a timeout (or delay) and a priority for the task queue (in this case MEDIUM priority). We report below an example of one of the task command:

void DishMaster::configure_rx_band_task(Tango::DevString argin)
{
  //=====================================================
  //==      CHECK & EXTRACT TASK DATA
  //=====================================================
  //- Extract argin encoded data to TaskData
  _DEBUG_LOG("Parsing argin json encoded TaskData...");
  std::string argin_json_str(argin);
  Utils_ns::TaskData taskData;
  if(Utils_ns::TaskUtils::ParseFromJsonString(taskData,this,argin_json_str)<0)
  {
    std::string errMsg("Failed to parse argin json string to TaskData!");
    _ERROR_LOG(errMsg);
    _THROW_TANGO_EXCEPTION("CONFIGURE_RX_BAND_TASK_FAILED",errMsg);
  }

  _DEBUG_LOG("Executing cmd="<<taskData.name<<", id="<<taskData.id<<", has_data? "<<taskData.has_data);

  //- Check if has argin data
  if(!taskData.has_data)
  {
    std::string errMsg("No argin data present (expecting a DevShort arg)!");
    _ERROR_LOG(errMsg);
    _THROW_TANGO_EXCEPTION("CONFIGURE_RX_BAND_TASK_FAILED",errMsg);
  }

  //- Extract argin task data
  Tango::DevShort bandId;
  if(taskData.GetData<Tango::DevShort>(bandId)<0)
  {
          std::string errMsg("Failed to extract data value in TaskData (check data type?)!");
    _ERROR_LOG(errMsg);
    _THROW_TANGO_EXCEPTION("CONFIGURE_RX_BAND_TASK_FAILED",errMsg);
  }

  _DEBUG_LOG("Executing command task "<<taskData.name<<" (id="<<taskData.id<<", has_data? "<<taskData.has_data<<", argin="<<bandId<<")");

  //=====================================================
  //==      EXECUTE COMMAND
  //=====================================================
  try {
    configure_rx_band(bandId);
  }
  catch(Tango::DevFailed& e){
    Tango::Except::print_exception(e);
    _ERROR_LOG("Tango exception occurred when commanding Rx band configuration!");
    throw;
  }
}

Event handling

A required feature for DishLMC is triggering callback commands to react on given registered events. For example according to given requirements we want to be able to execute actions on events like power cut, communication loss from TM or sub-elements, startup failures. Further we want also to handle events as soon as received as well as with a time delay (deferred handling). The latter situation occurs for the power cut event in which, by requirements, we are requested to set Single Pixel Feed (SPF) sub-element with a delay of 30 seconds to prevent unnecessary disruption in case power is restored soon.

To this aim we implemented an event handler registration feature and a priority queue mechanism to execute handler commands at desired timestamps or delay with respect to event receipt. Each registered handler task can be given a priority level (LOW, MEDIUM, HIGH) and an execution delay (0 means ‘executed immediately’). When the event is received the task is pushed in the task queue according to given specifications. When task queue is full events are not handled.

To register an event handler you must perform the following steps:

  1. Register the event (already described in previous section)

  2. Create an handler command in your device to be executed on event receipt. Handler commands must have a string attribute. For example:

    void DishMaster::handle_estop_event(Tango::DevString argin)
{
  //Extract task encoded json data from argin (see example in the Task section)
  //...

  //Execute task handler logic with extracted data
  //...
}

    NB: You should avoid handler commands lasting long times, because the next task in the queue will wait for previous task completion to execute. In this case inside your handler command use an asynchronous request mechanism or create a task sequence and push it to the task queue

  3. Register an event handler. You can do it in different ways:

    • Manual registration: Override the int RegisterEventHandlers() virtual method provided in the DishBaseDevice in your derived device and use the _REGISTER_EVT_HANDLERS(<proxy_name>,<attr_name>,<evt_type>,<evt_handlers>,<exec_delay>,<priority>) method. The example below registers in a device MyDevice an handler command MyHandlerCmd to change event for attribute rx_mode in proxy (mid_dsh_0001/rx/controller):

      int MyDevice::RegisterEventHandlers()
{
  std::string proxy_name("tango:://localhost:10000/mid_dsh_0001/rx/controller");
  long int executionDelay= 10000;//in ms
  int priority= Utils_ns::Task::eMEDIUM_PRIORITY;
  _REGISTER_EVT_HANDLERS(proxy_name,"rxOperatingMode",Tango::CHANGE_EVENT, {"MyHandlerCmd"}, executionDelay,int priority);

  //...

  return 0;

}//close RegisterEventHandlers()

  • Registration from properties: You can also specify proxies to be registered in the EventHandlers string list property. You must follow this format in the list:

    [0]= Full attribute name 1 (e.g. "tango:://localhost:10000/mid_dsh_0001/rx/controller/rxOperatingMode")
[1]= Event type 1 (e.g. "change", "periodic", "user", ...)
[2]= Command handler name 1 (e,g, "MyHandlerCmd")
[3]= Delay time in ms (0 means no delay)
[4]= Priority (0=LOW, 1=MEDIUM, 2=HIGH)
...
[N-4]= Full attribute name N
[N-3]= Event type N
[N-2]= Command handler name N
[N-1]= Delay time in ms
[N]= Priority N

Public API Documentation

Automatically generated with Doxygen from src/ska/tangods/DishBaseDevice

Warning

doxygenclass: Cannot find file: /home/docs/checkouts/readthedocs.org/user_builds/ska-telescope-ska-dish-lmc/checkouts/2.5.0/docs/src/api/xml/index.xml