How-to guides#

Task-oriented guides for working with core dumps.

Note

Core dumps are collected on the stfc-techops CI/CD and staging clusters only. See Where core dumps are collected for what is collected where.

Enable collection for your pods#

On stfc-techops production, the collector processes a crashed pod only if the pod carries the label coredump. The value is ignored — only the presence of the key matters. Pods without the label crash without leaving a dump.

Add it to the pod template, not the deployment metadata:

Deployment pod template#

spec:
  template:
    metadata:
      labels:
        coredump: enabled

For a Helm chart, add it under whichever values key the chart uses for pod labels, then confirm it landed on the running pod:

kubectl get pod <pod> --namespace <your-namespace> \
  -o jsonpath='{.metadata.labels}' | jq

Warning

A core dump holds the entire memory of the crashed process, including secrets it had loaded, tokens, and any user data in flight. Label only the workloads whose crashes you need to debug, and treat downloaded dumps as sensitive.

Find a core dump#

Open the Core Dumps Browser and sign in with GitLab.

Choose the Bucket for the environment the crash happened in.
Narrow the list with Hostname (the crashing pod’s name), Namespace, Date From, and Date To.
Click Search.

The results table shows the object key, UUID, crash time, pod name, namespace, executable, and size. Each row has a download action.

Filtering happens on the server, over the whole bucket listing, so a narrow namespace filter returns much faster than paging through everything. The dump metadata is parsed out of the object key itself — see Object key format.

Read the metadata without opening the dump#

The JSON documents in the zip answer most triage questions on their own, and they are far smaller than the core file.

Which signal killed it, in which pod, on which node:

jq '{exe, signal, real_pid, hostname, node_hostname, timestamp}' *-dump-info.json

Which image to reproduce against:

jq -r '.repoTags[0], .repoDigests[0]' *-image-info.json

Which deployment the pod belonged to, and how it was configured:

jq '.labels, .annotations' *-pod-info.json

What the container printed before dying:

tail -n 50 *-0.log

Analyse a dump with the SKAO script#

analyse.sh does the whole flow in one command. It unpacks the zip, prints the metadata, resolves the crashed executable and the container image digest from the JSON files, then runs gdb — and optionally pystack — inside that image and writes the backtraces to disk.

./analyse.sh core-dump.zip --docker --pystack --show-pystack

Option	Effect
`--docker`	Run the analysis inside the crashing container image, taken from the dump’s `repoDigests`
`--pystack`	Also run `pystack`, for Python processes
`--show-gdb` / `--show-pystack`	Print the report rather than only writing it to a file
`--no-gdb`	Skip `gdb` — useful for minimal images with no debugger
`--image <ref>`	Analyse in a different image, for example a debug build
`--binary <path>`	Load a different executable, for example one rebuilt with symbols
`--out <dir>`	Extract into a chosen directory instead of a temporary one
`--json`	Print the report locations as JSON

Reports land beside the extracted files as gdb-report.docker.txt and pystack-report.txt, or gdb-report.txt when run without --docker.

The script needs unzip, plus docker for --docker; it uses jq for metadata and falls back to python3. With --docker it runs the container as root with --cap-add=SYS_PTRACE and installs gdb and pystack inside it, so the image needs a package manager and network access. For a distroless or heavily stripped image, pass --image with a debug variant instead.

Use this for a first pass. The manual steps below are for when you need an interactive prompt.

Analyse a Python core dump#

pystack reconstructs Python-level tracebacks from a core file, including the C frames underneath.

Run it inside the same image the process crashed in, addressed by digest so you get byte-identical interpreter binaries:

docker run --rm -it -v "$PWD:/work" -w /work \
  $(jq -r '.repoDigests[0]' *-image-info.json) bash

pip install pystack
pystack core *.core

Useful options:

Command	Purpose
`pystack core <file>`	Python traceback for every thread
`pystack core --native <file>`	Interleave C/C++ frames with the Python frames
`pystack core --locals <file>`	Include local variables in each Python frame

Note

Analysis must run on the same architecture as the crash. SKAO worker nodes are x86_64, so on an Apple Silicon machine pass --platform linux/amd64 to docker run.

Analyse a compiled core dump#

For C and C++ processes, use gdb in the crashing image. The path field of *-dump-info.json gives the executable, with ! in place of /.

docker run --rm -it -v "$PWD:/work" -w /work \
  $(jq -r '.repoDigests[0]' *-image-info.json) bash

gdb /path/to/executable /work/<dump>.core

lldb works equally well if you prefer it and the image provides it.

At the gdb prompt:

(gdb) bt              # backtrace of the crashing thread
(gdb) thread apply all bt   # backtrace of every thread
(gdb) info registers
(gdb) frame 1         # select a frame, then inspect locals
(gdb) info locals

Backtraces are only readable if the binary carries debug symbols. Production images are usually stripped — rebuild the same commit with debug symbols and point gdb at that binary with the original core file.

Diagnose a missing core dump#

Work through these in order:

Check	What to do
Was the pod labelled?	On stfc-techops production the `coredump` label is mandatory. See Enable collection for your pods.
Did the process actually dump?	Only signals that dump core produce a file — `SIGSEGV` (exit code 139), `SIGABRT` (134), `SIGFPE` (136), `SIGBUS` (135). An `OOMKilled` pod, a non-zero `exit(1)`, or a `SIGTERM` during rollout leaves nothing behind.
Is the datacentre covered?	Only stfc-techops collects dumps today. Buckets exist for low-aa, low-itf, and mid-itf, but no collector runs there yet, so they stay empty. See Where core dumps are collected.
Is it older than 5 days?	Bucket lifecycle rules delete every object after 5 days. Nothing recovers an expired dump.
Is the process very large?	The composer gives up on a dump after 600 seconds. A multi-gigabyte process on a busy node can exceed that.
Is the collector healthy?	Ask the System Team, or check the DaemonSet in Headlamp — namespace `core-dump-handler`, one pod per covered node.