Usage in Pipelines

The Metric Generator is meant to be run with the processing pipelines as part of a visibility receive process.

The current generator config is as follows:

Note

The kafka host needs to be updated as required

Note

This config is meant to be appended to processing_blocks[].parameters.processors

{"signal-metrics": {
    "name": "qa-metrics-generator-plasma-receiver",
    "image": "artefact.skao.int/ska-sdp-qa-metric-generator",
    "version": "0.18.0",
    "command": [
        "plasma-processor",
        "ska_sdp_qa_metric_generator.plasma_to_metrics.SignalDisplayMetrics",
        "--plasma_socket",
        "/plasma/socket",
        "--readiness-file",
        "/tmp/processor_ready",
        "--use-sdp-metadata",
        "False",
        "--metrics",
        "all"
    ],
    "readinessProbe": {
        "initialDelaySeconds": 5,
        "periodSeconds": 5,
        "exec": {"command": ["cat", "/tmp/processor_ready"]}
    },
    "env": [
        {"name": "BROKER_INSTANCE", "value": "ska-sdp-kafka.test"}
    ]
}}

However, it is better to rather use the new style of config. You can also use this if you just want to override some of the values.

{"signal-display-metrics-all": {}}

The names that can be used is a follows:

• signal-display-metrics-all - this will run all the metrics
• signal-display-metrics-amplitude - this will only generate the amplitude graphs
• signal-display-metrics-basic - this will generate the base stats, and fast stats
• signal-display-metrics-phase - this will only generate the phase graphs

If you want to override the version you can do the following (any of the values can be overriden):

{"signal-display-metrics-all": {"version": "0.18.0"}}

The generator also outputs data on another Kafka topic used for monitoring. And to use that topic with the Queue Connector use the following config.

Note

The kafka host needs to be updated as required. And the topic name contains the subarray ID

Note

This config is meant to be appended to processing_blocks[].parameters.queue_connector_configuration.exchanges

{
    "dtype": "object",
    "shape": [],
    "source": {
        "type": "KafkaConsumerSource",
        "servers": "ska-sdp-kafka.test",
        "topic": "metrics-receive_state-01",
        "encoding": "json",
    },
    "sink": {
        "type": "TangoObjectScatterAttributeSink",
        "attributes": [
            {
                "attribute_name": "receiver_state",
                "filter": "type=='visibility_receive'",
                "path": "state",
                "dtype": "str",
                "default_value": "unknown",
            },
            {
                "attribute_name": "last_update",
                "filter": "type=='visibility_receive'",
                "path": "time",
                "dtype": "float",
                "default_value": 0.0,
            },
            {
                "attribute_name": "processing_block_id",
                "filter": "type=='visibility_receive'",
                "path": "processing_block_id",
                "dtype": "str",
                "default_value": "",
            },
            {
                "attribute_name": "execution_block_id",
                "filter": "type=='visibility_receive'",
                "path": "execution_block_id",
                "dtype": "str",
                "default_value": "",
            },
            {
                "attribute_name": "subarray_id",
                "filter": "type=='visibility_receive'",
                "path": "subarray_id",
                "dtype": "str",
                "default_value": "-1",
            },
            {
                "attribute_name": "scan_id",
                "filter": "type=='visibility_receive'",
                "path": "scan_id",
                "dtype": "int",
                "default_value": 0,
            },
            {
                "attribute_name": "payloads_received",
                "filter": "type=='visibility_receive'",
                "path": "payloads_received",
                "dtype": "int",
                "default_value": 0,
            },
            {
                "attribute_name": "time_slices_received",
                "filter": "type=='visibility_receive'",
                "path": "time_slices",
                "dtype": "int",
                "default_value": 0,
            },
            {
                "attribute_name": "time_since_last_payload",
                "filter": "type=='visibility_receive'",
                "path": "time_since_last_payload",
                "dtype": "float",
                "default_value": 0.0,
            },
        ],
    },
}