How it works#

Understand how core dumps are captured, stored, and retrieved at SKAO.

Why core dumps#

When a process dies from a fault such as an invalid memory access, the interpreter or runtime rarely gets the chance to write anything useful to its logs. In Kubernetes the evidence disappears faster still: the container is gone, its filesystem with it, and a restart policy may have replaced it before anyone notices.

A core dump is the process’s memory captured at the instant it died. From it you can recover the call stack of every thread, the local variables in each frame, and the C-level frames underneath interpreted code.

From crash to download#

Collection happens on the Kubernetes node, outside your container:

  1. Your process faults. The kernel hands its memory to a collector running on the node rather than writing a file into the container.

  2. The collector packages the memory image together with metadata about the pod it came from into a single zip file.

  3. The zip is uploaded to the S3 bucket for that environment and removed from the node.

  4. The Core Dumps Browser lists it, so you can filter by namespace, pod, and date, and download it.

To enable this on each pod, you must label it with coredump: enabled. See Enable collection for your pods for instructions.

Why pods must opt in#

On stfc-techops production, only pods carrying the coredump label have their crashes captured. The label is a deliberate opt-in for three reasons.

Dumps are large, and a crash-looping pod produces them continuously. Every dump is a full memory image carrying whatever secrets and user data the process held at the time, so capturing them indiscriminately carries significant risk. , so it is important not to capture them indiscriminately.

Labelling the workloads you want covered keeps the volume proportionate and makes capturing memory an explicit choice. See Enable collection for your pods.

Why dumps expire after 5 days#

Core dumps are large and briefly useful: they matter while an incident is open, and after that they become a growing store of memory images nobody reads. Every bucket therefore expires objects after 5 days.

The consequence is that retrieval is time-sensitive. Download a dump while you are still investigating since an expired one cannot be recovered.

What never produces a dump#

Only faults the kernel treats as dumping signals reach the collector — SIGSEGV, SIGABRT, SIGBUS, SIGFPE, and SIGILL. Nothing is captured when:

  • The container is OOMKilled, since the kernel terminates it rather than faulting it.

  • The process exits non-zero on its own, however badly.

  • The pod is stopped with SIGTERM during a rollout or a scale-down.

  • The pod carries no coredump label in an environment that requires one.

  • The pod runs in a datacentre with no collector deployed. Only stfc-techops collects dumps today — see Where core dumps are collected.

For a crash that leaves no dump, fall back to Monitoring and logging for logs and metrics, and Debugging for live inspection.

Handle a downloaded dump#

Treat the file as sensitive. It contains everything the process had in memory — credentials it had read, tokens it was holding, and any user data in flight. Keep it on trusted storage, do not attach it to tickets or paste it into chat, and delete local copies when the investigation closes.

Access to the buckets themselves is never handed out. The Internal Developer Platform holds those credentials and streams objects through its own API, so downloading a dump follows your authenticated platform session.

Was this page helpful?