Tutorials

Learn where to find and access contextualised monitoring dashboards, logs, and debugging tools for your deployments.

What you’ll learn

By the end of this tutorial, you’ll have:

• Followed the monitoring and logging links from a pipeline job and a merge request

• Reached namespace-scoped monitoring and logging from Headlamp

• Opened Kibana and searched the logs for a namespace

• Opened Grafana and found the dashboards for a deployment

• Downloaded logs and artefacts from a pipeline job

• Found the deployment information for a running namespace

Time required: 30–45 minutes

Prerequisites:

• A GitLab pipeline that has deployed to a Kubernetes namespace, so that it has produced links to follow

• Network access to the monitoring and logging services for that datacentre — see Service URLs

Pipeline links

When you run a CI/CD pipeline, links to monitoring and logging resources appear in the GitLab job output. These links include your job’s metadata and time range.

Note

To enable these links in your project, integrate with the pipeline machinery.

General links in GitLab pipeline output

Log links in GitLab pipeline output

Monitoring links in GitLab pipeline output

These links include:

• Headlamp URL for the namespace

• Pipeline log files for all namespace pods and pod descriptions

• Kibana links for namespace logs, test pods, and Device Server configuration

• Grafana links for namespace metrics, Device Servers, and compute dashboards

Tip

Copy and paste Kibana URLs into the browser — clicking them may not work due to URL length.

Merge request links

The same links appear in your merge requests, organised by job with deployment type, namespace, and target cluster.

Monitoring and logging links in merge requests

Headlamp links

In Headlamp, use the top-right buttons to reach namespace-scoped monitoring and logging pages. Use these as starting queries to refine further. The page also gives direct access to namespace-scoped and resource-scoped metrics.

Monitoring and logging links in Headlamp

Access Kibana for logs

1. Go to Kibana

2. Click Continue as Guest for read-only access

3. Navigate to Discover in the left pane

4. Select your time range and add filters for your namespace

For full access (creating API keys, saving searches), raise a Kibana account request. See Set up Kibana access in the how-to guide for the full walkthrough.

Access Grafana for metrics

1. Go to Grafana

2. Click Sign in with Azure AD

3. Use your JIRA credentials: <jira-username>@<community.skao.int or assoc.skao.int>

4. Browse existing dashboards or create your own

See Use Grafana dashboards in the how-to guide for browsing, creating, and sharing dashboards.

Browsing Grafana dashboards

Useful dashboards

CI/CD pipelines:

• CI/CD Pipeline Jobs

• CI/CD Pipeline Status

Namespace resources:

• Namespace Resources

• Namespace Overview

• Namespace Usage

Workloads:

• Device Servers

• Compute Resources by Workload

• Compute Resources by Pod

Usage analytics (Kibana):

• Kubernetes namespace share by Team, Project and User

Usage analytics (Headlamp):

• Kubernetes namespace share by Team

• Kubernetes namespace share by User

• Kubernetes namespace share by Project

Download pipeline artefacts

Pipeline jobs include log artefacts you can download for offline analysis:

curl -L https://gitlab.com/ska-telescope/ska-ser-oci-daemon/-/jobs/9819118142/artifacts/download -o job.zip
mkdir -p artefacts && cd artefacts
unzip ../job.zip
ls -R

This downloads:

• Kubernetes events — All events in the namespace

• Pod descriptions — Output of kubectl describe for every pod

• Pod logs — Logs from all pods

Pipeline job artefacts structure

View deployment information

Pipeline output shows deployed images and Helm charts:

***Gathering information for namespace: ci-ska-tango-examples-7cabaa1f***
OCI images for pod databaseds-tangodb-tango-databaseds-0:
    artefact.skao.int/ska-tango-images-tango-db:11.0.2

Installed Helm charts:
   test:
      Chart: ska-tango-examples-test-parent-0.1.16
      App Version: 0.1.16
   Dependencies:
      * ska-tango-base @ 0.4.16 | 0.4.18 Available!
      * ska-tango-util @ 0.4.16 | 0.4.18 Available!

The output shows installed versions and flags when newer versions are available.

This information also appears in Headlamp’s namespace view:

Helm release information in Headlamp

Next steps

• Read How-to guides for detailed log filtering and Elasticsearch queries

• Check Reference for the SKA Log Message Format, log search fields and service URLs

• Explore How it works for Headlamp and Coder debugging tools