Reference#

Coverage, buckets, object key format, configuration variables, and API endpoints for the SKAO core dump service.

Where core dumps are collected#

The collector runs as a DaemonSet in the core-dump-handler namespace of the clusters below. Datacentres that are absent from this table produce no core dumps.

Datacentre	Bucket	Pod label required	Browse at
stfc-techops (production)	`stfc-techops-production-core-dumps`	`coredump`	idp.devx.skao.int/core-dumps
low-aa (production)	`low-aa-production-core-dumps`	`app`	idp.devx.skao.int/core-dumps
low-itf (production)	`low-itf-production-core-dumps`	`app`	idp.devx.skao.int/core-dumps
mid-itf (production)	`mid-itf-production-core-dumps`	`app`	idp.devx.skao.int/core-dumps

Storage and retention#

Property	Value
Provider	AWS S3, region `eu-west-2`
Retention	5 days — a lifecycle rule expires every object in every bucket
Versioned objects	Non-current versions under the `ns-gitlab-hn-` prefix expire after 1 day (production techops bucket)
Access from the cluster	One IAM user per bucket, `coredumps-<bucket>-s3-bucket-user`, limited to `GetObject`, `PutObject`, `ListBucket`, and `DeleteObject` on that bucket
Access for developers	Through the Core Dumps Browser only — the bucket credentials stay server-side

Buckets, IAM users, policies, and access keys are provisioned by the s3-core-dumps Terraform stack in ska-ser-infra-machinery.

Warning

Nothing recovers a dump after 5 days. Download anything you still need to investigate.

Object key format#

SKAO overrides the chart’s default filename template so that the object key itself carries the searchable metadata. The Core Dumps Browser parses the key to populate its columns and filters.

ns-{namespace}-hn-{hostname}-en-{exe_name}-ts-{timestamp}-uuid-{uuid}.zip

Field	Browser column	Meaning
`{namespace}`	Namespace	Kubernetes namespace of the crashed pod
`{hostname}`	Hostname	Name of the crashed pod, not the node
`{exe_name}`	Executable	Name of the crashed executable, for example `python`
`{timestamp}`	Datetime	Unix seconds at crash time
`{uuid}`	UUID	Identifier the composer generates per dump

Zip contents#

Every file inside the zip shares the key’s base name, with the suffixes below.

Suffix	Contents
`.core`	The core dump — full process memory image
`-dump-info.json`	`exe`, `path`, `real_pid`, `signal`, `hostname` (pod), `node_hostname`, `timestamp`, `uuid`, `dump_file`
`-image-info.json`	Image ID, `repoTags`, `repoDigests`, size — use the digest to reproduce the runtime
`-pod-info.json`	Pod labels, annotations, namespace, UID, and sandbox state from CRI
`-ps-info.json`	Container list with IDs, image refs, and state at crash time
`-runtime-info.json`	Full container runtime description
`-<attempt>.log`	Tail of the container log, up to `logLength` lines

In dump-info.json, the path field encodes the executable path with ! in place of / — !usr!local!bin!python3.12 means /usr/local/bin/python3.12.

Signals that produce a core dump#

Signal	Number	Pod exit code	Typical cause
`SIGSEGV`	11	139	Invalid memory access
`SIGABRT`	6	134	`abort()`, uncaught C++ exception, failed assertion
`SIGBUS`	7	135	Misaligned or unmapped memory access
`SIGFPE`	8	136	Arithmetic fault such as integer division by zero
`SIGILL`	4	132	Illegal instruction

OOMKilled pods, ordinary non-zero exits, and pods stopped with SIGTERM produce no core dump.

External documentation#

analyse.sh — SKAO script that unpacks a dump and produces backtraces
core-dump-handler on GitHub — upstream chart and agent
core-dump-handler chart documentation
core-dump-handler Helm repository
pystack — Python traceback recovery from core files
core(5) — kernel core dump behaviour and core_pattern