Problem RestatementProblem
When a process crashes, the OS can write a core dump: a snapshot of its memory at the moment of the crash, which engineers use to debug. Design a system (asked at Amazon) that collects core dumps from a large fleet of hosts, stores them, and helps engineers analyze them: which crashes are new, how often they happen, and on which versions.
Core dumps can be huge (GBs), a bad deploy can cause thousands of crashes at once, and dumps can contain sensitive data (memory may include secrets or customer data).
RequirementsRequirements
- Capture dumps on crash, without filling the host disk.
- Upload reliably to central storage, with limits during crash storms.
- Extract metadata: service, version, host, time, signal, and the stack trace (after symbolication).
- Group crashes by signature (same bug → same group), and count occurrences.
- Search, download (for authorized engineers) and link to tickets.
- Alert on new crash signatures, and on spikes after deploys.
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
P["Crashing process"] --> H["Host agent - core handler"]
H --> SP[("Local spool - size capped")]
SP -->|"compressed, resumable upload"| UP["Upload service - rate limited"]
UP --> OS[("Object storage - encrypted dumps")]
UP --> Q[("Processing queue")]
Q --> AN["Analyzer - symbolicate, extract stack"]
SYM[("Symbol store by build ID")] --> AN
AN --> DB[("Crash DB - metadata, signatures, groups")]
DB --> UI["Crash dashboard"]
DB --> ALR["Alerts - new signature, spike"]Key Steps
- Capture: configure the kernel to pipe core dumps to our agent (
core_patternwith a pipe), instead of writing wherever. The agent writes a compressed file (zstd) into a spool directory with a size cap. If the disk is almost full, it keeps a minidump (just stacks and registers) instead of the full dump. - Quick local metadata: service name, binary build ID, version, host, signal, timestamps.
- Upload: resumable multipart upload to object storage (encrypted). The upload service applies rate limits and sampling: during a crash storm, keep full dumps for the first N per signature per hour, and just counts and minidumps for the rest.
- Analyze: a worker fetches the dump, loads debug symbols for that build ID from the symbol store (uploaded by CI at build time), produces a symbolicated stack trace, and computes a signature (e.g., a hash of the top 5 meaningful frames, ignoring addresses and line numbers that change between builds).
- Group and store: add to the crash group for that signature, updating counts, first seen, last seen and affected versions.
Deep Dive — Turning thousands of crashes into a handful of bugsDeep dive
A fleet produces crashes continuously. The raw stream is unusable; the value is entirely in grouping crashes that share a cause.
One alert per crash
Every core dump notifies the owning team.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
BUG["One null-pointer bug ships"] --> C["40,000 crashes in an hour"]
C --> A["40,000 alerts"]
A --> MUTE["The team mutes the channel"]
MUTE --> MISS["A second, unrelated crash goes unnoticed"]Volume destroys signal. The predictable outcome is that alerting gets muted, and then the system's real job — telling you something new is broken — stops working entirely.
Group by the top stack frame
Use the crashing function as the group key and alert once per group.
Volume collapses to something readable, and it is the right instinct. But the top frame is often not the bug: thousands of unrelated crashes all end in abort, malloc, or an allocator assert, so genuinely different bugs merge into one useless group. The inverse happens too — inlining and template instantiation mean one bug produces several different top frames and splits into several groups.
A normalised signature, and alert on what changed
Build the signature from several frames, after cleaning them up:
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
ST["Stack trace"] --> SKIP["Skip generic frames - abort, malloc, assert handlers"]
SKIP --> NORM["Normalise - strip template args, collapse inlined frames, drop addresses"]
NORM --> SIG["Signature - top N meaningful frames"]
SIG --> GRP[("Crash group")]
GRP --> NEW{"New signature in production?"}
NEW -->|"yes"| OWN["Notify the owning team"]
GRP --> RATE{"Crash rate per version jumped?"}
RATE -->|"yes"| DEPLOY["Alert and link the deploy - suggest rollback"]- Skip the generic frames so different bugs stop merging, and normalise names and addresses so one bug stops splitting. Both directions matter, and only fixing one makes grouping worse in the other.
- Alert on novelty and on rate, not on volume. A new signature in production is worth waking someone for. A known crash at its usual rate is not, however many times it happens.
- Key the rate by version and link it to the deploy system. "Crash rate for this signature tripled in v4.12" turns an alert into an action — roll back — which is the only thing anyone can do at 3 a.m. anyway.
The dashboard then ranks by count with a trend and the affected versions, and each group links to example dumps and its ticket, so the one-crash-to-one-bug mapping holds all the way to the fix.
Security and Retention
- Dumps may contain secrets or customer data: encrypt at rest, restrict download to the owning team with audit logs, and prefer analysis in a secure environment (engineers view stacks, not raw memory, unless approved).
- Retention: keep full dumps for 14–30 days (a few examples per group longer), and keep metadata and stacks for a long time. Delete by lifecycle policy.
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Capture | Pipe to agent with capped spool | Protects host disk | Default core files: disks fill up |
| Crash storms | Sample per signature + minidumps | Bounded storage and bandwidth | Upload all: overload during incidents |
| Analysis | Central symbolication by build ID | Readable stacks | Raw addresses: useless for humans |
| Grouping | Normalized top-frame signature | Tracks bugs over time | Group by error message: too coarse |
Wrap-UpWrap-up
Route core dumps to a host agent that compresses them into a size-capped spool (falling back to minidumps), uploads them resumably with rate limiting and per-signature sampling, and records metadata. Central analyzers symbolicate stacks using a build-ID symbol store, compute normalized signatures, and group crashes, which powers dashboards and alerts on new signatures and post-deploy spikes, with encryption, access control and lifecycle retention for sensitive memory contents.