Reference

Log format standards, field reference, and service URLs.

SKA Log Message Format

All processes running in containers must log to stdout.

Log messages must conform to this format for ingestion:

VERSION "|" TIMESTAMP "|" SEVERITY "|" [THREAD-ID] "|" [FUNCTION] "|" [LINE-LOC] "|" [TAGS] "|" MESSAGE LF

Field definitions:

Field	Required	Description
VERSION	Yes	Version of SKA log standard (1-2 digits, starts at 1)
TIMESTAMP	Yes	ISO8601 timestamp in UTC (e.g., 2019-12-31T23:12:37.526Z)
SEVERITY	Yes	Log level: DEBUG, INFO, WARNING, ERROR, or CRITICAL
THREAD-ID	No	Thread identifier (e.g., “MainThread”, “Thread-1”)
FUNCTION	No	Full namespace of function (e.g., package.module.Class.method)
LINE-LOC	No	Filename and line number (e.g., test.py#150)
TAGS	No	Comma-separated key:value pairs (e.g., tango-device:my/dev/name)
MESSAGE	Yes	UTF-8 encoded message content

Format examples

1|2019-12-31T23:12:37.526Z|INFO||testpackage.testmodule.TestDevice.test_fn|test.py#1|tango-device:my/dev/name| Regular information logged here
1|2019-12-31T23:45:42.328Z|DEBUG||testpackage.testmodule.TestDevice.test_fn|test.py#150|| x = 67, y = 24
1|2019-12-31T23:49:53.543Z|WARNING||testpackage.testmodule.TestDevice.test_fn|test.py#16|| z is unspecified, defaulting to 0!
1|2019-12-31T23:50:17.124Z|ERROR||testpackage.testmodule.TestDevice.test_fn|test.py#165|site:Element| Could not connect to database!
1|2019-12-31T23:51:23.036Z|CRITICAL||testpackage.testmodule.TestDevice.test_fn|test.py#16|| Invalid operation. Cannot continue.

Logging levels

Map Python logging levels to RFC5424 (syslog):

Python	RFC5424	Numerical Code
DEBUG	Debug	7
INFO	Informational	6
WARNING	Warning	4
ERROR	Error	3
CRITICAL	Critical	2

Parsing strategies

Split by delimiter:

log_line = "1|2019-12-31T23:50:17.124Z|ERROR||my.module.func|test.py#165||Cannot connect to database"
structured_log = log_line.split('|')
log_level = structured_log[2]   # "ERROR"

Regex with named capture:

^(?<version>\d+)[|](?<timestamp>[0-9TZ\-:.]+)[|](?<level>[\w\s]+)[|](?<thread>[\w-]*)[|](?<function>[\w\-.]*)[|](?<lineloc>[\w\s.#]*)[|](?<tags>[\w\:,-]*)[|](?<message>.*)$

Test at: https://rubular.com/r/e0njVOGCN59mtA

Log field reference

Use these fields to filter logs in Kibana or query Elasticsearch locally.

Kubernetes fields:

Field	Description
kubernetes.namespace	Kubernetes namespace
kubernetes.pod.name	Pod name
kubernetes.statefulset.name	StatefulSet name (useful for Device Servers)
kubernetes.container.name	Container name
kubernetes.node.name	Node name

SKA infrastructure fields:

Field	Description
ska.datacentre	Datacentre (e.g., stfc-techops, mid-itf)
ska.environment	Environment (e.g., production)
ska.application	Log source (syslog, journald, docker, podman, kubernetes)

SKA CI/CD fields (prefix with kubernetes.labels. or kubernetes.namespace_labels.):

Field	Description
cicd_skao_int/projectId	GitLab project ID
cicd_skao_int/project	GitLab project name
cicd_skao_int/projectPath	Sanitised GitLab project path
cicd_skao_int/author	Author name
cicd_skao_int/authorId	Author GitLab ID
cicd_skao_int/team	SKA team (from People’s database)
cicd_skao_int/commit	Commit SHA
cicd_skao_int/branch	Branch name
cicd_skao_int/pipelineId	GitLab pipeline ID
cicd_skao_int/jobId	GitLab job ID
cicd_skao_int/job	GitLab job name
cicd_skao_int/mrId	Merge request ID (if applicable)
cicd_skao_int/environmentTier	GitLab environment tier
cicd_skao_int/pipelineSource	Pipeline trigger source

SKA custom log fields:

Field	Description
ska_severity	Log severity level
ska_tags_field.<tag>	Dynamic log message tags

Log parsing pipelines

Before Elasticsearch indexes a log, the document passes through an ingest pipeline that turns raw log text into the structured, searchable fields below and strips sensitive data. Two pipelines run at SKAO, both defined in the ska-ser-ansible-collections repository.

SKA log parsing pipeline

The ska_log_parsing_pipeline grok-parses the pipe-delimited SKA Log Message Format from the message and log fields and produces:

Field	Source	Description
ska_version	message / log	SKA log format version
ska_log_timestamp	message / log	Log timestamp (ISO8601, UTC)
ska_severity	message / log	Log level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
ska_thread_id	message / log	Thread identifier
ska_function	message / log	Fully-qualified function name
ska_line_loc	message / log	Filename and line number
ska_tags	message / log	Raw comma-separated key:value tags
ska_message	message / log	The log message body
ska_tags_field.<key>	ska_tags	Each tag expanded into its own field (e.g. ska_tags_field.tango-device)
ska	ska_message	Nested object parsed from a JSON ska_message
ska_transaction_id	message / log	Transaction id extracted from Transaction[txn-local-...]
is_audit_log	derived	true for host bash_history audit logs
is_redaction_exempt	derived	true only for an allow-listed SDP application tagged redact:false
ska_pipeline	derived	Set to ska_log_parsing_pipeline to record which pipeline ran

Redaction: To meet GDPR and security requirements, the pipeline removes email addresses, IPv4 addresses, Bearer tokens, passwords and API keys, passkeys, curl -u credentials, connection-string credentials, usernames, and GitLab user metadata from container labels — across the message, log, and ska_message fields. Host bash_history audit logs (is_audit_log) are exempt so the audit trail stays intact.

Kubernetes audit pipeline

The ska_k8s_audit_pipeline parses Kubernetes audit.k8s.io JSON events and maps them onto common fields:

Field	Source (audit JSON)	Description
kubernetes.namespace	objectRef.namespace	Target namespace
kubernetes.resource	objectRef.resource	Resource type
kubernetes.name	objectRef.name	Resource name
kubernetes.api_version	objectRef.apiVersion	API version
event.action	verb	API verb (get, create, delete, …)
user.name	user.username	User that made the request
source.ip	sourceIPs[0]	Source IP address
http.response.status_code	responseStatus.code	HTTP response code
url.original	requestURI	Requested URI

Standard tags

Use these tags in log messages for filtering:

deviceName:

TANGO device name in format <facility>/<family>/<device>.

Example: MID-D0125/rx/controller

• MID-D0125 — Dish serial number

• rx — Dish Single Pixel Feed Receiver (SPFRx)

• controller — Dish SPFRx controller

subSystem:

For non-TANGO software, the telescope sub-system name.

Valid values: CSP, Dish, INAU, INSA, LFAA, SDP, SaDT, TM

Service URLs

Services vary by datacentre. Production environments do not expose every service.

Grafana monitoring for every datacentre is at monitoring.skao.int.

Logging services — CI/CD datacentres:

Datacentre	Kibana	Elasticsearch
stfc-techops	Open	https://logging.stfc.skao.int:9200
stfc-dp	Open	https://logging.stfc.skao.int:9200

Logging services — ITF and production datacentres:

Datacentre	Kibana	Elasticsearch
mid-itf, low-itf	Open	https://logging.stfc.skao.int:9200
mid-aa	Open	https://logging.mid.internal.skao.int:9200
low-aa	Open	https://logging.low.internal.skao.int:9200
aws-*	Open	https://logging.stfc.skao.int:9200

Note

When STFC is down for maintenance, find logging on aws-cicd in Kibana.

Central logging filter values — CI/CD datacentres:

Datacentre	ska.datacentre	ska.environment	ska.prometheus_datacentre
stfc-techops	stfc-techops	production	stfc-ska-monitor
stfc-dp	stfc-dp	production	stfc-dp-ska-monitor

Central logging filter values — ITF and production datacentres:

Datacentre	ska.datacentre	ska.environment	ska.prometheus_datacentre
mid-itf	mid-itf	production	mid-itf-monitor
low-itf	low-itf	production	low-itf-monitor
psi-mid	psi-mid	production	psi-mid-monitor
digital-signal-psi	digital-signal-psi	production	N/A

External documentation

• Elasticsearch Documentation

• Kibana Documentation

• ES|QL Syntax

• KQL Reference

• Kubernetes Log Fields

• Grafana Dashboard Guide

• Prometheus Documentation

• Python Logging HOWTO