Problem RestatementProblem
Design a platform (asked at Capital One) that ingests events from producers in multiple regions, stores them durably, and delivers them to multiple consumers for processing. For example, transaction events consumed by fraud detection, notifications and analytics. It must survive a whole region failing without losing events, keep ordering per key (e.g., per account), and let consumers replay past events.
RequirementsRequirements
- Producers send events continuously from several regions.
- Durable storage: no acknowledged event is lost, even if a region goes down.
- Multiple independent consumer groups, each tracking its own progress.
- Ordering per key (per account ID).
- At-least-once delivery, with support for consumers to deduplicate.
- Replay from a point in time. Low latency (sub-second in region).
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
PA["Producers - Region A"] --> KA[("Kafka cluster A")]
PB["Producers - Region B"] --> KB[("Kafka cluster B")]
KA <-->|"async replication - MirrorMaker / cluster linking"| KB
KA --> CA["Consumers in A - fraud, notifications"]
KB --> CB["Consumers in B"]
CA --> DED[("Dedup store / idempotent sinks")]
CB --> DED
REG["Schema registry"] --- PA
REG --- PB- Per-region Kafka clusters (or a managed equivalent) with replication factor 3 across availability zones in the region, and
acks=allso a write is on multiple brokers before being acknowledged. - Cross-region replication copies topics between regions (asynchronously) so the other region has the data if one fails.
- Partitioning by key (account_id) keeps per-key order within a partition.
- Schema registry for event formats and compatibility.
Design Choices
3.1 Active-passive vs active-active
- Active-passive: all producers write to Region A, and it's replicated to B. On failure, switch producers and consumers to B. Simpler ordering, but a failover step is needed, and events not yet replicated can be lost (RPO = replication lag).
- Active-active (our choice for availability): producers write to their local region. Each region has "local" topics plus "mirrored" topics from the other region. Consumers that need everything read both. Per-key ordering is kept by routing each key to a home region (e.g., an account's events always go to its home region's topic), with failover of the home region when needed.
3.2 Durability vs latency
- Synchronous cross-region writes (a "stretch cluster") give RPO = 0 but add cross-region latency (tens of ms) to every write. Use this only for critical events, like money movement.
- Asynchronous replication is faster, with a small RPO (seconds). State it and choose per topic.
Deep Dive — How many times does an event get processed?Deep dive
Producers in several regions, consumers doing fraud checks, notifications and analytics. What "delivered" means has to be decided explicitly, because every option is wrong in some way.
Commit the offset, then process
The consumer marks the event consumed as soon as it reads it, then does the work.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
R["Read event, commit offset"] --> P["Process"]
P --> CRASH["Worker crashes mid-processing"]
CRASH --> GONE["Offset already committed - event never retried"]
GONE --> LOST["A transaction is never fraud-checked, and nothing reports it"]This is at-most-once, and the loss is silent. For a transaction stream that is the worst possible failure mode: no error, no gap, just a record that quietly never got processed.
Process, then commit the offset
Flip the order: do the work, then commit. Nothing is lost, because a crash before the commit means the event is re-delivered.
At-least-once, which is the right guarantee — and it means duplicates are now normal, not exceptional. A re-delivered event sends a second notification, or double-counts in analytics. The guarantee is correct and the consumers are not yet ready for it.
At-least-once, with consumers that do not care
Leave delivery at at-least-once and make every side effect safe to repeat:
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
K[("Event log - event_id on every event")] --> C1["Fraud consumer - upsert by event_id"]
K --> C2["Notification consumer - dedup store with TTL"]
K --> C3["Analytics consumer - idempotent aggregate write"]
C1 --> CM["Commit offset after processing"]
C2 --> CM
C3 --> CM
CM -->|"crash before commit"| RE["Re-delivered - second attempt is a no-op"]- A unique
event_idon every event, assigned by the producer. Without it the consumer has no way to recognise a repeat. - Idempotent writes where the store allows it — upsert keyed by
event_idrather than insert or increment. - A dedup store with a TTL where it does not. The notification consumer records the ids it has acted on for long enough to cover the retry window, which is hours, not forever.
One clarification worth making unprompted: Kafka's exactly-once applies to read-process-write inside Kafka. It cannot make an external side effect — an email, a card being charged, a row in another database — happen exactly once. If the work leaves the system, idempotency at the consumer is the only mechanism that works, and claiming otherwise is a common way to lose the thread of a design interview.
Region Failover
- Region A goes down. Producers switch to B (clients configured with both endpoints, or through DNS/traffic management).
- Consumers in B continue from their offsets. For the mirrored copy of A's topics, offsets differ between clusters, so use offset translation (checkpointing the mapping, as MirrorMaker 2 does) or resume by timestamp, accepting some duplicates (dedup handles them).
- When A returns, reverse-sync, then fail back gradually.
Operations
- Monitor producer error rates, replication lag per topic (this is effectively your RPO), consumer lag per group, and broker health.
- Keep retention long enough for replays (e.g., 7 days), with tiered storage to object storage for longer.
- Run regular failover drills.
Wrap-UpWrap-up
Run a replicated Kafka cluster per region (acks=all across zones), partition by key for ordering, and replicate topics across regions: asynchronously by default (small RPO) and synchronously only where zero loss is required. Prefer active-active with a home region per key. Deliver at-least-once with event IDs and idempotent consumers, handle failover with offset translation or timestamp resume, and monitor replication and consumer lag while regularly practicing failover.