Problem RestatementProblem
Google asked: design a pub/sub system that supports replay. Besides normal real-time delivery (publishers send, subscribers receive new messages), a subscriber must be able to rewind: "re-deliver everything since yesterday 9:00" (e.g., after fixing a bug that processed messages wrongly), or seek to a specific position. This is like Google Pub/Sub "seek" or Kafka's offset reset.
Key Idea: A Durable Log, Not a Queue
A classic queue deletes messages once acknowledged, so there's nothing to replay. Instead, store messages in an append-only log kept for a retention period (e.g., 7 days). Each subscriber just tracks its position (offset) in the log. Replay = move the position back.
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
PUB["Publishers"] --> BR["Brokers - topic partitions"]
BR --> LOG[("Partition logs - segments on disk")]
LOG -->|"old segments"| TIER[("Tiered storage - object store")]
BR --> TIDX[("Time index - timestamp to offset")]
SUB["Subscribers / consumer groups"] -->|"fetch from offset"| BR
SUB --> OFF[("Committed offsets per subscription")]
ADM["Seek API"] --> OFF- Topics → partitions: messages with the same key go to the same partition, which keeps order per key.
- Each partition is a log of segments (files). Each message has an offset and a timestamp.
- Time index per segment: maps timestamps to offsets, so "replay from 9:00" is a fast lookup.
- Subscriptions store committed offsets per partition.
Replay FlowFlows
- The operator calls
seek(subscription, timestamp=yesterday 09:00)(or an offset or a snapshot). - For each partition, find the first offset with timestamp ≥ 09:00 using the time index (binary search), and set the subscription's position there.
- Consumers resume fetching from the new position. Old segments may come from tiered storage (slower, but cheap).
- Messages are re-delivered in order within each partition.
Deep Dive — Replaying history without hurting live deliveryDeep dive
A subscriber wants to reprocess the last three days after fixing a bug. Live consumers and publishers must not notice.
Replay from the live brokers
Let the replaying subscriber read old messages from the same brokers serving real-time traffic.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
RP["Replay - 3 days of data"] --> BR["Live brokers"]
BR --> DISK["Sequential reads of cold segments"]
DISK --> CACHE["Page cache evicted - live reads now hit disk"]
CACHE --> SLOW["Real-time consumers fall behind"]
BR --> PUB["Publishers see higher latency"]The replay is a large scan of cold data on machines tuned for hot, recent data. It evicts the page cache that live delivery depends on, so one subscriber's backfill degrades everyone — including the publishers.
Bound it with a retention window
Keep messages for a fixed period and allow replay within it.
This makes replay possible and bounds storage, which is necessary. It does nothing about the interference: a replay inside the window still reads through the live brokers, and the retention you can afford on broker disks is short — often too short for the incident you are recovering from.
Tiered storage, a separate read path, and per-subscription limits
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
PUB["Publishers"] --> LOG[("Durable log - recent segments on broker disks")]
LOG --> LIVE["Live consumers - hot path"]
LOG -->|"aged out"| TIER[("Tiered storage - object storage, weeks or months")]
RP["Replay request - seek to time or snapshot"] --> READ["Historical read path"]
TIER --> READ
READ --> RL["Per-subscription rate limit"]
RL --> SUB["Replaying subscriber"]
SNAP["Named snapshot of a subscription's positions"] --> RP- Tiered storage decouples retention from broker disks. Old segments move to object storage, so months of replay become affordable and the brokers keep only what live delivery needs.
- A separate read path means historical reads are served from object storage rather than competing for the brokers' page cache and IO.
- Rate-limit per subscription, so one enthusiastic replay cannot consume the historical read capacity everyone else shares.
- Named snapshots of subscription positions taken before a risky deploy turn "seek to roughly Tuesday" into "seek to exactly where we were" — which is what makes recovery precise rather than approximate.
The requirement that makes all of it usable: consumers must be idempotent. Replay means reprocessing messages that were already processed, so unless the side effects are safe to repeat, the feature creates a second incident while recovering from the first.
Wrap-UpWrap-up
Store every topic as partitioned, append-only logs with offsets, timestamps and a time index, keep them for a retention window (extended cheaply with tiered storage), and let each subscription track its own committed positions. Replay is a seek: map a timestamp or snapshot to per-partition offsets and resume fetching, in order per partition. Protect live traffic with read isolation and rate limits, and require idempotent consumers so reprocessing is safe.