How-to guides

Practical tasks for working with logs and metrics.

Note

Examples use central logging URLs. For production datacentres (low-aa, mid-aa), use the service URLs listed in the reference.

Set up Kibana access

Guest access (read-only):

Go to Kibana and click Continue as Guest.

Full access:

1. Raise a Kibana account request

2. Log in with your JIRA username and the password from the STS response

3. Create an API key: Stack Management → API keys. You can customise the permissions of the key.

Creating an API key in Kibana

Test your API key:

curl -k -H "Authorization: ApiKey <your-api-key>" \
  https://logging.stfc.skao.int:9200/_cat/health

Filter logs in Kibana

Kibana’s Discover view provides log searching with these controls:

1. Data view — Select index patterns

2. Date selection — Set time range

3. Filters — Filter by fields with AND/OR operators

4. Search bar — KQL expressions (combined with filters using AND)

5. Field list — Available fields to display

6. Document view — Matching log entries

Kibana Discover view

To find Device Server logs in a specific namespace:

Filtering logs by namespace

Use filters for exact matches and KQL for substring searches. Each filter shows the most common values:

Field value suggestions

Export and share log queries

Once you have a useful search in Discover, you can share it, take the query elsewhere, or export the matching logs.

Share a link to your search

A shared link reproduces your exact data view, filters, time range, and query — the same mechanism behind the prebuilt pipeline links surfaced in CI/CD jobs.

1. Build your query in Discover.

2. Click Share in the top bar, then Link → Copy link.

3. Send the link. Kibana links are long — the recipient must copy and paste it into the browser rather than clicking it.

Sharing a Discover search as a link

Take the query to curl or elktail

To run the same search outside Kibana, copy the underlying query:

1. Switch the query bar to ES|QL, or open Inspect → Request to see the raw Elasticsearch query.

2. Copy it and adapt it for a local query — see Query Elasticsearch locally and the elktail section below.

Inspecting the raw Elasticsearch request behind a search

Export matching logs to CSV

1. Add the fields you want as columns in the Discover document table (the CSV contains the displayed columns).

2. Click Share → Export, then Generate CSV in the Export Discover session as CSV panel.

3. When the report is ready, click Download report in the notification, or fetch it later from Stack Management → Reporting.

Generating a CSV export of a Discover search

Downloading the generated CSV report

Note

Large exports are subject to Kibana Reporting size limits. For bulk or automated extraction, query Elasticsearch directly (below).

Query Elasticsearch locally

For efficient log analysis, query Elasticsearch directly using ES|QL.

Find logs by namespace and component:

API_KEY=<your-api-key>
curl -qk -X POST "https://logging.stfc.skao.int:9200/_query?format=json&pretty" \
  -H "Authorization: ApiKey $API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "query": "FROM filebeat-8* | WHERE ska.datacentre == \"stfc-techops\" AND ska.environment == \"production\" AND kubernetes.namespace == \"ska-ser-namespace-manager\" AND `kubernetes.labels.app_kubernetes_io/component` == \"api\" | KEEP message | LIMIT 100"
  }' 2>/dev/null | jq -r ".values[][]"

Find logs by GitLab job ID:

curl -qk -X POST "https://logging.stfc.skao.int:9200/_query?format=json&pretty" \
  -H "Authorization: ApiKey $API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "query": "FROM filebeat-8* | WHERE ska.datacentre == \"stfc-techops\" AND ska.environment == \"production\" AND `kubernetes.labels.cicd_skao_int/jobId` == \"10006002600\" | KEEP message | LIMIT 1000"
  }' 2>/dev/null | jq -r ".values[][]"

Sort by oldest first:

Add SORT @timestamp ASC to the query.

Filter by time range:

Add WHERE @timestamp > TO_DATETIME("2025-05-13T05:00:00Z") AND @timestamp <= NOW().

Filter by container name:

Add WHERE kubernetes.container.name LIKE "test-runner*".

Tip

Build queries in Kibana’s ES|QL mode first, then convert to curl commands.

Enable ES|QL in Kibana

Use elktail for command-line queries

elktail provides command-line log queries using KQL syntax.

Installation:

Download binaries from the releases page.

Configuration:

Generate a config file using this snippet, then replace the APIKey value.

Basic query:

elktail -n 1 ska.application: syslog AND ska.datacentre: mid-itf AND message: dnsmasq

Query container logs:

elktail -n 1 "ska.datacentre: mid-itf AND kubernetes.namespace: staging-dish-lmc-ska100 AND kubernetes.statefulset.name: ds-dish-logger-100 AND kubernetes.container.name: deviceserver"

Custom output format:

elktail -n 1 -F "%ska_log_timestamp :: SEV: [%ska_severity] MSG -> %ska_message" \
  "ska.datacentre: mid-itf AND kubernetes.namespace: staging-dish-lmc-ska100"

Inspect raw JSON:

elktail -p -n 1 ska.application: syslog AND ska.datacentre: mid-itf

Use Grafana dashboards

Because Grafana shows metrics and logs on the same dashboard and time range, it is the best place to correlate the two — lining up a log line with the CPU or memory spike it belongs to, so you can move from symptom to root cause in a single view.

Access Grafana:

1. Go to https://monitoring.skao.int

2. Click Sign in with Azure AD

3. Use <jira-username>@<community.skao.int or assoc.skao.int> with your JIRA password

Browse dashboards:

Browsing Grafana dashboards

Create and share dashboards:

Follow the Grafana dashboard guide to create custom dashboards.

Sharing a Grafana dashboard

View logs in Grafana

Several Grafana dashboards include integrated log views:

Pod logs in Grafana

The log timeframe adjusts automatically to match the dashboard selection, so you can scrub to a spike on a metric panel and immediately read the log lines emitted at that moment — matching, say, an error message to the CPU or memory spike right beside it. You can filter using the same fields available in Kibana:

Log fields in Grafana

Check platform alerts

Prometheus Alert Manager (VPN required):

• STFC: http://monitoring.techops.internal.skao.int:9093/#/alerts

• DP: http://monitoring.sdhp.skao:9093/#/alerts

Prometheus Alert Manager

Grafana OnCall (no VPN required):

Grafana OnCall unifies alert management for the STFC and DP datacentres in the central monitoring platform. View and manage alerts in the OnCall app.

Grafana Oncall

Slack alert channels:

Alerts from Grafana OnCall are posted to #alerts-techops-oncall.

Per-datacentre channels:

Datacentre	Platform alerts	Developer alerts
stfc-techops	#techops-alerts	#techops-user-alerts
stfc-dp	#dp-platform-alerts	#dp-platform-user-alerts
aws-*	N/A	#aws-eu-west-2-user-alerts

Monitor namespace health

The SKA Namespace Manager monitors namespace health and sends Slack notifications when health degrades or namespaces are deleted:

Namespace health alerts from Marvin

These messages include the reason, affected resources, suggestions, and runbooks. They also link to the GitLab job that deployed the namespace.

You can also use the datacentre’s Headlamp (when available) to inspect the health status of the namespace and its resources:

Namespace status in Headlamp