Problem RestatementProblem
Bloomberg asked two related questions:
- Multiple exchanges (NASDAQ, NYSE, NSE, ...) send market data to loader servers. The loaders distribute several copies of each message over UDP to processor servers. Design this pipeline so that processors see every message exactly once and in order, quickly.
- Build a real-time VWAP provider. VWAP (volume-weighted average price) = Σ(price × volume) ÷ Σ(volume) over a window. It's the average price actually traded, weighted by size. Compute it for thousands of symbols from high-volume trade ticks, and publish it to subscribers.
RequirementsRequirements
- Ingest trades and quotes from many exchanges (millions of messages/sec at peaks).
- Deliver to processors with very low latency (microseconds to low milliseconds).
- No lost or duplicated messages, and the per-symbol order is preserved.
- VWAP per symbol for windows (e.g., since market open, and rolling 5 minutes), updated in real time.
- Subscribers get updates for the symbols they care about.
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
EX1["Exchange A feed"] --> L1["Loader A"]
EX2["Exchange B feed"] --> L2["Loader B"]
L1 -->|"UDP multicast - copy 1"| P["Processors - partitioned by symbol"]
L1 -->|"UDP multicast - copy 2"| P
L2 -->|"UDP multicast - copies"| P
P --> V["VWAP state per symbol"]
V -->|"updates"| PUB["Publisher / fan-out"]
PUB --> S["Subscribers - terminals, apps"]
L1 --> RR[("Retransmit / replay store")]
P -->|"gap request"| RRDeep Dive — Getting every message, in order, off UDP multicastDeep dive
Multicast is used because one packet reaches every processor at wire speed. The price is that UDP loses, duplicates and reorders packets, and a market data feed cannot tolerate any of the three.
Trust the multicast
Processors read packets and apply them as they arrive.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
L["Loader - UDP multicast"] --> P["Processor"]
P --> D["Duplicate packet applied twice"]
D --> VW["Volume double-counted - VWAP wrong"]
P --> LOST["Packet dropped by a switch buffer"]
LOST --> MISS["Trade missing - VWAP silently wrong all session"]
P --> OOO["Reordered packets - cancel applied before its trade"]Every one of these corrupts a running total silently. There is no error and no way to notice: the VWAP is simply a number that no longer matches the market, and it stays wrong for the rest of the session.
Sequence numbers and gap detection
Every message from a loader carries an increasing seq per channel. The processor tracks the last sequence it applied: a repeat is a duplicate and gets dropped, a jump means a gap. On a gap, ask a retransmit service for the missing messages and buffer the later ones briefly to preserve order.
This is correct, and recovery is the slow path. A single lost packet stalls that symbol while a request goes out and comes back — milliseconds during which the price is not updating, which on a fast-moving symbol is exactly when it matters.
Two independent paths, arbitrated by sequence
Send every message twice, over two independent network paths, and let the processor take whichever copy arrives first.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
LD["Loader"] --> A["Path A - multicast"]
LD --> B["Path B - multicast"]
A --> ARB["Arbitrator - first copy of each seq wins"]
B --> ARB
ARB --> APP["Apply in sequence order"]
ARB -->|"missing on both paths"| RTX["Retransmit service - recent message cache"]
RTX --> APP
APP -->|"recovery too slow"| STALE["Mark the symbol stale - never publish a wrong price"]
SNAP["Snapshot channel"] --> JOIN["Restarting or late processor loads state, then applies seq after it"]A/B arbitration hides essentially every single-path loss with no delay at all — the other copy has usually already arrived, and deduplication by sequence number is what makes taking both safe. Retransmission drops back to being the rare case, for a message genuinely lost on both paths.
Two rules that belong in the answer:
- A stale price is acceptable; a wrong price is not. If recovery is too slow, mark the symbol stale and stop publishing rather than continuing with a total you know is missing a trade.
- Snapshots for joining late. A processor that restarts loads a snapshot of current state and then applies messages after the snapshot's sequence number. Without this, a restart means replaying the whole session or starting blind.
Computing VWAP
For each symbol, keep running sums:
since-open VWAP: pv_sum += price × qty; v_sum += qty; vwap = pv_sum / v_sum
rolling 5-min: keep per-second buckets (pv, v) in a ring of 300; add new, drop expired;
vwap_5m = Σ bucket.pv / Σ bucket.v- It's O(1) per trade and O(1) per update (keep running totals of the ring too).
- Use fixed-point integers for prices (e.g., price × 10^4) to avoid floating-point drift.
- Handle trade corrections and cancels from exchanges by subtracting the original trade's contribution.
- Partition by symbol: each processor owns a set of symbols, so all ticks for a symbol go to one processor in order, with no locks. Hot symbols (AAPL, TSLA) get dedicated cores.
Publishing to Subscribers
- Subscribers subscribe to symbols. The publisher sends VWAP updates on change, conflated (e.g., at most every 100 ms per symbol per subscriber). Slow subscribers get the latest value, not a backlog.
- Many subscribers → fan-out through a tier of distribution servers (multicast inside the data center, TCP/WebSocket to outside clients).
Latency and Failover
- Low latency: kernel-bypass networking, pinned CPU cores, no garbage-collection pauses in the hot path (C++/Rust or tuned Java), and in-memory state only.
- Failover: run hot-standby processors that consume the same multicast and compute the same state. If the primary dies, the standby takes over instantly with identical state (deterministic processing of the same ordered input).
- Normalization: each exchange has its own format and symbol codes. Loaders convert to one internal format and symbol ID.
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Transport | UDP multicast with A/B copies | Lowest latency, efficient fan-out | TCP: reliable but slower and per-receiver |
| Loss handling | Seq numbers + arbitration + retransmit | Complete, ordered stream | Ignore gaps: wrong prices |
| Parallelism | Partition by symbol | Ordered, lock-free per symbol | Shared state: locks and contention |
| Output | Conflated updates | Handles slow consumers | Send every tick: overload |
Wrap-UpWrap-up
Loaders normalize exchange feeds and send each message with sequence numbers over two multicast paths. Processors partitioned by symbol take the first copy, drop duplicates, detect gaps and recover them from a retransmit store, and load snapshots when they (re)join. VWAP is kept as running sums (since open) and per-second ring buckets (rolling windows) in fixed-point math, published to subscribers as conflated updates, and hot-standby processors keep identical state for instant failover.