Diagnose a Kubernetes workload with Headlamp#

Use Headlamp to investigate an unhealthy namespace or workload through a web interface. Headlamp shows only resources permitted by your Kubernetes role.

Important

The ITF and Array Assembly environments — mid-itf, low-itf, mid-aa, and low-aa — do not expose Headlamp on a public URL. Reach it through InfraHQ instead. Set this up before you need it, because an incident is the worst time to discover you have no access. See Access a cluster with InfraHQ.

Headlamp opens on a cluster-wide workload summary: how many Pods, Deployments, StatefulSets, and Jobs are running, and how many have failed. Use it to tell a cluster-wide problem from one confined to your namespace before you start digging.

Headlamp home view showing cluster-wide counts of running and failed Pods, Deployments, StatefulSets, and Jobs above a filterable workload table.

The Headlamp home view, summarising workload health across the cluster.#

Open the namespace#

  1. Open the Headlamp URL for the target environment from Service URLs.

  2. Sign in and select the target cluster when Headlamp presents a cluster selector.

  3. Open Namespaces and select the namespace that contains the workload.

  4. Check the namespace health summary and identify resources that are not ready.

Tip

A CI/CD pipeline emits a Headlamp URL for the namespace it deployed, alongside its logging and monitoring links. Following that link opens the namespace directly and skips the first three steps. See Pipeline links.

Headlamp namespace page highlighting healthy and unhealthy resources.

Inspect namespace health in Headlamp.#

The same page carries the SKAO CI/CD metadata for the deployment:

  • the GitLab project the namespace was deployed from

  • the team that owns it

  • the person who deployed it

  • links to the originating pipeline and job

That tells you who to contact and what produced the namespace before you have finished diagnosing the problem, which matters most when the namespace belongs to another team. Trace the deployment uses the same metadata to identify the change that introduced the fault.

When you do not know which namespace to open, use the Organisation view. It groups namespaces by SKAO team, user, and project, so you can find every namespace your team has deployed without knowing its name. Reconstructing this from labels by hand is slow, and the plain Kubernetes views do not present it.

Headlamp organisation view summarising namespace usage grouped by SKAO team.

Find namespaces by team, user, or project.#

Inspect the workload#

Open the relevant Deployment, StatefulSet, or Pod. Check its status, conditions, restart count, container images, resource requests, and recent events. Events expose common causes such as failed scheduling, image pull errors, missing Secrets, and failed health probes.

For a controller, follow its relationship to the affected Pod. For a Pod with multiple containers, select each container in turn and compare their states.

Read the logs#

Open the Pod’s log view and select the affected container. Start with the time around the latest restart or failed probe. Search for the first error in a chain rather than only the final exception.

Headlamp Pod page with the log viewer open, showing the container selector, line count, and the Previous, Timestamps and Follow toggles above SKA-format log lines.

Read container logs beside the Pod’s metrics.#

The viewer’s controls matter for a restarting Pod:

  • Container selects which container in the Pod you are reading

  • Previous shows the logs of the terminated instance, which is where the cause of a restart usually is

  • Timestamps and Follow let you correlate against events, or watch a Pod as it starts

  • Lines raises the tail length when the error scrolled past

Logs written in the SKA Log Message Format carry the severity, function, line, and TANGO device in each line, so you can identify the failing device from the log view without opening another tool.

Check the metrics#

The namespace page carries a metrics panel from the SKAO Prometheus plugin. Switch between CPU, memory, network, and filesystem, and change the time range and resolution, without leaving Headlamp. A memory curve that climbs to a plateau as a Pod starts restarting is the kind of correlation this panel makes obvious immediately.

Headlamp namespace page showing the built-in memory metrics panel, with CPU, network and filesystem tabs, a time-range selector, and contextual icons in the top right.

Read namespace metrics without leaving Headlamp.#

The icons in the top right of the namespace page open the corresponding monitoring and logging views, already scoped to that namespace. Use them when you need retained logs, a longer time range, or a full dashboard.

Trace the deployment#

Return to the CI/CD metadata on the namespace page. Follow the branch, commit, pipeline, and job links to identify the deployment that introduced the running version, and compare it against the last deployment known to work.

Headlamp namespace page showing SKAO CI/CD ownership and deployment metadata.

Trace a namespace back to its GitLab deployment.#

Open the Helm releases view to compare the installed chart against its dependency versions. A workload running an unexpected dependency version is often the difference between a working namespace and a broken one, and this view is the quickest way to see it.

Headlamp Helm releases view listing installed charts and their dependency versions in a namespace.

Compare installed chart and dependency versions.#

Inspect a TANGO deployment#

Where the workload uses the SKA Tango Operator, the SKAO Tango plugin adds views for its custom resources. The TANGO section opens on a summary across every namespace you can see: how many device servers and DatabaseDS instances are running against how many are still waiting, the StatefulSets behind them, and the CRD, operator, and ping versions in use. A device server stuck in Waiting is visible here without opening a namespace at all.

Headlamp TANGO overview showing CRD, operator and ping versions, counts of running and waiting device servers, DatabaseDS instances and StatefulSets, above a table of device servers with their namespaces and status.

Device servers and DatabaseDS instances across the cluster.#

Filter by namespace, or open the namespace’s own Device Servers view, to narrow the same table to one deployment. Each row links the device server to the StatefulSet the operator created for it, so you can move from a TANGO name to the running Pod in one step.

Headlamp device server table scoped to a single namespace, listing each device server with its device count, status, and backing StatefulSet.

Device servers in a single namespace.#

Open a device server to see its labels, annotations, image version, dependencies, and the StatefulSet running it. The Config control opens the device server’s dsconfig as the operator applied it, with the classes, properties, and attribute properties it was given. Comparing that against what the device is meant to have is the quickest way to find a misconfiguration, and it saves querying the TANGO database by hand.

Continue in the Coder workflow when you need tango_admin or an ITango session against the database.

Choose the next action#

Use the evidence from status, events, logs, metrics, and deployment metadata to identify the owning team and likely change. Headlamp respects Kubernetes RBAC; an unavailable action can mean that your role is read-only rather than that the cluster is malfunctioning.

Move to the Coder workflow when you need terminal commands, in-cluster DNS or network tests, or an IDE. Use the How-to guides for detailed Kibana and Elasticsearch log searches.

Upstream documentation#

This guide covers the SKAO deployment. Headlamp itself is an open-source Kubernetes web interface, and its own documentation describes the standard views this page builds on.

The SKAO plugins that add organisation, metrics, ownership, Helm, and Tango views are listed in Headlamp SKAO views.

Was this page helpful?