Problem RestatementProblem
Design a system (asked at Atlassian) that takes a high-throughput stream of events (e.g., product usage events), validates and transforms them, computes aggregations over time windows (e.g., active users per workspace per minute), and writes the results to downstream sinks (a database, dashboard store or another topic). It must handle out-of-order and late events, recover from crashes without losing or double-counting, and scale out.
RequirementsRequirements
- Ingest 100K+ events/sec.
- Validate (schema, required fields), drop or quarantine bad events.
- Transform and enrich (e.g., add workspace plan from a lookup).
- Windowed aggregations (tumbling 1-minute, sliding 5-minute) per key.
- Output within seconds. Correct counts after crashes.
- Scale horizontally.
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
SRC["Producers"] --> K[("Kafka - partitioned by key")]
K --> V["Validate + parse"]
V -->|"bad"| DLQ[("Quarantine topic")]
V --> E["Enrich - cached lookups"]
E --> W["Windowed aggregate - keyed state"]
W --> SINK[("Sink - DB / OLAP / topic")]
W --> CK[("Checkpoints - object storage")]Built on a stream framework like Flink or Kafka Streams, which provides keyed state, windows, watermarks and checkpoints.
Key Concepts (in simple words)
- Partitioning: events are keyed (e.g., by workspace_id). All events for a key go to the same partition and the same processing task, so per-key order holds and state is local.
- Event time vs processing time: event time = when it happened on the device, and processing time = when we see it. Aggregating by event time gives correct results even if events arrive late.
- Watermark: the processor's estimate that "all events up to time T have arrived" (e.g., the max seen event time minus 30 seconds). A window closes when the watermark passes its end. Later events can update the result (allowed lateness) or go to a side output.
- State: running counts per key and window, stored in the operator (e.g., RocksDB on local disk).
- Checkpointing: periodically snapshot all state plus the Kafka offsets together. After a crash, restore the snapshot and re-read from those offsets. No loss, no double counting inside the processor.
- Exactly-once to the sink: use idempotent upserts (key = window + key) or transactional sinks committed with checkpoints.
Flow ExampleFlows
Event {workspace: w1, user: u9, ts: 12:00:42} → validated → enriched with plan = "premium" → added to the w1 window [12:00, 12:01) set of users → at watermark 12:01:30 the window emits {w1, 12:00, active_users: 57} → upsert into the metrics DB.
Deep Dive — When the sink is slower than the streamDeep dive
Events arrive at a rate you do not control. The warehouse is having a bad afternoon. What the pipeline does in that gap is the difference between a delay and a data loss incident.
Buffer in memory and keep reading
Keep pulling from the source and hold what the sink has not accepted yet.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
SRC["Events - full rate"] --> BUF["In-memory buffer"]
BUF --> SINK["Sink - degraded, half speed"]
BUF --> GROW["Buffer grows for the whole outage"]
GROW --> OOM["Task killed - everything buffered is gone"]
OOM --> RESTART["Restarts, reads at full rate, fills again"]An unbounded buffer converts a slow sink into an out-of-memory crash, and the crash loses precisely the data that was waiting. Then it repeats, because nothing about the restart changed the rate mismatch.
Bound the buffer and drop
Cap the buffer and discard the oldest events when it fills. The process survives.
Memory is safe and the pipeline keeps running, which is better than crashing. But dropping is a silent, permanent data loss, and it happens during the incident — exactly when the aggregates are most likely to be looked at. For usage events that feed billing or product metrics, that is not an acceptable trade.
Push back, and let the durable log be the buffer
Slow down reading from the source instead of buffering or dropping:
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
K[("Kafka - retention hours to days")] --> OP["Operator"]
OP --> SINK["Sink - slow"]
SINK -->|"slow"| BP["Backpressure propagates upstream"]
BP --> OP
OP -->|"reads more slowly"| K
K --> LAG["Consumer lag grows - the visible signal"]
LAG --> ALERT["Alert on lag, not on buffer size"]
K --> CATCH["Sink recovers - pipeline drains the backlog"]Kafka is already a durable, disk-backed buffer with hours of retention, so the right move is to let data wait there rather than in the process's heap. Nothing is dropped, nothing is at risk in memory, and recovery is automatic once the sink returns.
Three things that make this work in practice:
- Alert on consumer lag. Lag is the honest measure of "we are behind", and it is the only one visible from outside the process.
- Pre-aggregate hot keys. One enormous workspace will saturate a single partition regardless of backpressure. Add a random sub-key, aggregate, then combine — otherwise one key sets the pipeline's throughput.
- Do not call a database per event for enrichment. Cache reference data locally or load it as a broadcast stream; a per-event lookup makes the pipeline's throughput a function of someone else's database.
Parallelism is bounded by partitions per operator, so scaling out means adding partitions as well as tasks — adding tasks alone leaves them idle.
Operations
- Monitor lag, throughput, checkpoint duration, late-event counts and quarantine rates.
- Reprocessing: to fix a bug, deploy a new job version that reads from an earlier Kafka offset (or from the data lake), and writes to a new table version, then switch.
- Schema evolution: a schema registry with backward-compatible changes.
Wrap-UpWrap-up
Read keyed events from Kafka, validate (quarantining bad ones), enrich with cached lookups, and aggregate in event-time windows closed by watermarks. Keep state in the operators with periodic checkpoints of state plus offsets for crash-safe exactly-once processing, write idempotently to sinks, scale by partitions (pre-aggregating hot keys), and rely on backpressure and lag monitoring to stay healthy.