Tutorials#

Learn to capture and inspect process core dumps from SKAO services.

A core dump is a snapshot of a process’s memory at the moment it crashed. SKAO collects core dumps from Kubernetes worker nodes automatically and stores them in S3, where you can search and download them through the Internal Developer Platform.

In this tutorial you deploy a pod that crashes on purpose, find its core dump in the Core Dumps Browser, and read the crashing Python traceback out of it.

What you’ll learn#

By the end of this tutorial, you’ll have:

Labelled a pod so that the collector captures its crash
Deployed a pod that segfaults on purpose
Confirmed the crash from the container’s exit code and signal
Found the resulting dump in the Core Dumps Browser
Unpacked the dump and read its crash metadata
Recovered the crashing Python traceback with pystack

Time required: 30-45 minutes

Before you start#

You need:

kubectl access to your own namespace on the stfc-techops CI/CD cluster. See Diagnose a Kubernetes workload with Headlamp for cluster access.
A GitLab account to sign in to the Internal Developer Platform.
Docker (or Podman) on your machine, to run the analysis container.

Note

Core dumps expire after 5 days. Complete the tutorial in one sitting, or repeat step 2 to generate a fresh dump.

Step 1 - Label the pod for collection#

On stfc-techops production, the collector only processes pods that carry the label coredump. Any value works; the collector checks only that the label key is present. Pods without it crash silently and no dump is stored.

You apply this label in the pod spec in the next step. For your real workloads, add it to the pod template of the deployment or Helm chart you want covered.

Warning

Add the label only where you want crash memory captured. A core dump contains the full memory of the process, including anything it had read into memory — credentials, tokens, and user data.

Step 2 - Deploy a crashing pod#

This pod calls ctypes.string_at(0), which dereferences a null pointer and raises SIGSEGV.

pysegfault.yaml#

apiVersion: v1
kind: Pod
metadata:
  name: python-segfault
  labels:
    coredump: enabled
spec:
  restartPolicy: Never
  containers:
    - name: python
      image: python:3.12-slim
      command:
        - /bin/sh
        - -c
        - |
          python - << 'EOF'
          import ctypes

          def myfunc():
              ctypes.string_at(0)

          print("About to segfault...")
          myfunc()
          EOF

Apply it to your own namespace:

kubectl apply -f pysegfault.yaml --namespace <your-namespace>

Step 3 - Confirm the crash#

kubectl get pod python-segfault --namespace <your-namespace> \
  -o jsonpath='{.status.containerStatuses[*].state}' | jq

{
  "terminated": {
    "containerID": "containerd://f48ca5f32a9ff63ec1596e9885e7ce53e5bdff1b5b9ca4a0773c1d7c8cbdfc76",
    "exitCode": 139,
    "finishedAt": "2025-12-22T14:08:38Z",
    "reason": "Error",
    "startedAt": "2025-12-22T14:08:38Z"
  }
}

Exit code 139 is 128 + 11, so the process died on signal 11 — SIGSEGV. That is what triggers the kernel to write a core dump.

Step 4 - Find the dump in the browser#

Open the Core Dumps Browser and sign in with GitLab. Then:

Set Bucket to stfc-techops-production-core-dumps.
Type your namespace into Namespace.
Click Search.

Your dump appears as a row with the crash time, the pod name under Hostname, and python under Executable.

Note

Hostname is the crashing pod’s name, not the node’s. The node name is recorded inside the dump metadata as node_hostname.

Upload takes a few seconds after the crash. If the row is missing, see Diagnose a missing core dump.

Step 5 - Download and unpack#

Click the download icon at the end of the row to fetch the zip file. Its name is the S3 object key, which encodes the namespace, pod, executable, and crash time:

unzip ns-<your-namespace>-hn-python-segfault-en-python-ts-1766412518-uuid-ee9a0f74-9b49-40fa-ba9c-79ff0b1f13b3.zip

You get the core file plus a set of JSON documents describing the crash, all sharing that same base name:

ns-...-uuid-<uuid>.core              # the core dump itself
ns-...-uuid-<uuid>-dump-info.json    # executable, pid, signal, pod and node hostname
ns-...-uuid-<uuid>-image-info.json   # container image ID and digest
ns-...-uuid-<uuid>-pod-info.json     # pod labels, annotations, namespace
ns-...-uuid-<uuid>-ps-info.json      # container state at crash time
ns-...-uuid-<uuid>-runtime-info.json # container runtime details
ns-...-uuid-<uuid>-0.log             # container log tail

Read the crash summary:

jq . *-dump-info.json

{
  "dump_file": "ns-my-namespace-hn-python-segfault-en-python-ts-1766412518-uuid-ee9a0f74-9b49-40fa-ba9c-79ff0b1f13b3.core",
  "exe": "python",
  "hostname": "python-segfault",
  "node_hostname": "stfc-techops-k8s-md-0-hhg8n-s4hpd",
  "path": "!usr!local!bin!python3.12",
  "real_pid": "8",
  "signal": "11",
  "timestamp": "1766412518",
  "uuid": "ee9a0f74-9b49-40fa-ba9c-79ff0b1f13b3"
}

Signal 11 confirms SIGSEGV. In path, the ! characters stand in for / — the executable was /usr/local/bin/python3.12.

Step 6 - Read the traceback#

A core dump is only readable against the same binaries the process ran. Take the image digest from *-image-info.json:

jq -r '.repoDigests[0]' *-image-info.json

Start that exact image with the dump files mounted:

docker run --rm -it -v "$PWD:/work" -w /work \
  docker.io/library/python@sha256:fa48eefe2146644c2308b909d6bb7651a768178f84fc9550dcd495e4d6d84d01 \
  bash

Inside the container, install pystack and point it at the core file:

pip install pystack
pystack core *.core

Using executable found in the core file: /usr/local/bin/python

Core file information:
state: R zombie: True niceness: 0
pid: 8 ppid: 1 sid: 1
uid: 0 gid: 0 pgrp: 1
executable: python arguments: python -

The process died due receiving signal SIGSEGV
Traceback for thread 8 [Has the GIL] (most recent call last):
    (Python) File "<stdin>", line 5, in <module>
    (Python) File "<stdin>", line 2, in myfunc
    (Python) File "/usr/local/lib/python3.12/ctypes/__init__.py", line 525, in string_at
        return _string_at(ptr, size)

The traceback names the crashing function, the line it died on, and the C call underneath it.

Tip

The analyse.sh script automates every command in steps 5 and 6 — unpacking, metadata, image resolution, and the backtrace. Now that you have seen what those commands do, use the script on real crashes: see Analyse a dump with the SKAO script.

Step 7 - Clean up#

kubectl delete pod python-segfault --namespace <your-namespace>

The stored dump expires on its own after 5 days.

Next steps#

How-to guides — analyse compiled binaries with gdb, filter the browser, and diagnose missing dumps.
Reference — buckets, object key format, and configuration variables.
How it works — how the collector hooks into the kernel and what it installs on each node.