How it works#

Understand the architecture and debugging tools available at SKAO.

Centralised architecture#

SKAO’s monitoring and logging solution aggregates data from all datacentres into a central system. This eliminates the need for multiple dashboards and provides:

  • Unified access — Single entry point for all environments

  • Data aggregation — Combine metrics and logs from multiple clusters

  • Zero-trust security — Secure communication between monitoring systems

  • Consistent tooling — Same tools work across all environments

Monitoring stack#

Prometheus collects metrics from Kubernetes clusters and infrastructure. It integrates with Thanos for high availability and long-term storage, enabling metric aggregation from multiple Prometheus instances.

Grafana provides dashboards for visualising metrics. Developers use pre-built dashboards for CI/CD pipelines, namespace resources, and workload monitoring, and can create and share custom dashboards.

Metrics flow from the scrape targets through to dashboards and alerts:

        flowchart LR
    K8s["Kubernetes clusters"] -->|"scrape"| P["Prometheus"]
    Infra["Infrastructure VMs"] -->|"scrape"| P
    P -->|"HA + long-term storage"| T["Thanos"]
    T --> G["Grafana dashboards"]
    P -->|"alert rules"| A["Alertmanager / Grafana OnCall"]
    A --> S["Slack"]

Logging stack#

Elasticsearch stores and indexes log data. Logs flow from containers through Filebeat agents to the central Elasticsearch cluster. On ingestion, each document is processed — where applicable — by an ingest pipeline that parses the SKA log format into structured fields and redacts sensitive data; see Log parsing pipelines for the fields produced.

Kibana provides the frontend for searching and visualising logs. While Kibana excels at creating visualisations, direct Elasticsearch queries using ES|QL or KQL are more efficient for log analysis.

Logs flow from container output through to Kibana and Grafana:

        flowchart LR
    C["Container stdout"] -->|"Filebeat"| IP["Ingest pipeline<br/>(parse + redact)"]
    IP --> ES[("Elasticsearch")]
    ES --> K["Kibana"]
    ES --> G["Grafana log panels"]
    ES -.->|"ES-QL / curl"| L["Local queries"]

Logs contain rich metadata including Kubernetes labels, CI/CD information, and SKA-specific fields that enable precise filtering.

Live debugging tools#

Headlamp complements retained monitoring data with a web view of current Kubernetes state. Coder provides a remote workspace for command-line and IDE investigations from inside selected clusters. BinderHub builds reproducible notebook environments from Git repositories.

Read Choose a SKAO debugging tool to choose between the tools and use the debugging service URL table to find the service available in each environment.

Namespace Manager#

The SKA Namespace Manager actively monitors namespace health. It evaluates each namespace every minute and notifies owners via Slack when issues occur.

Marvin Namespace Manager

Marvin alerting via Slack#

Alerts include:

  • Affected resources

  • Suggestions for resolution

  • Links to runbooks

  • Links to the deployment job

This helps identify unhealthy namespaces that may be consuming resources without active use. For more details on the Namespace Manager, check the project documentation here.

Log format design#

The SKA log format balances several goals:

Minimum useful information:

  • Timestamp for ordering and correlation

  • Log level for filtering by severity

  • Extensible tags for arbitrary metadata

  • Source location for debugging

Easy parsing:

The pipe-delimited format allows simple splitting by delimiter. The fixed number of fields (8 in version 1) enables predictable parsing.

Readability:

Human-readable format for local development without requiring JSON parsing.

Versioning:

The version field allows format evolution. A new version number indicates changes to fields after the first delimiter.

Why not syslog (RFC5424)?

Containerised applications log to stdout, where the container runtime handles routing. The ingestion layer can transform logs to syslog format if needed. The SKA format prioritises container-native patterns and developer ergonomics.

Data retention#

Monitoring and logging data have retention limits:

  • Logs — Available for a defined retention period (varies by environment)

  • Metrics — Long-term storage via Thanos, but older data may have reduced granularity

For longer-term log access:

  • Download pipeline artefacts (available as long as the job exists)

  • Use Coder for direct cluster access (bypasses retention limits)

Pipeline links include the deployment timeframe, so queries return nothing after the retention period expires.

Was this page helpful?