Problem RestatementProblem
Design a brokerage app where users see live stock prices, place buy and sell orders, and track their orders, holdings and balance. The broker does not match trades itself. It sends orders to a stock exchange (like NSE or NASDAQ), which matches buyers and sellers and reports back fills (executions).
Key challenges:
- Never let a user spend money they don't have.
- Keep every order's status correct, even when messages from the exchange are delayed or repeated.
- Survive the huge spike when the market opens.
RequirementsRequirements
1.1 Functional
- Live quotes for stocks.
- Place orders: market (buy now at the best price), limit (buy only at ₹100 or lower), stop-loss. Modify and cancel orders.
- Show order status: open, partially filled, filled, cancelled, rejected.
- Show holdings, P&L and available funds.
1.2 Non-Functional
- Correctness of money and positions above everything.
- Low latency from order click to exchange (tens of milliseconds inside our system).
- High availability during market hours.
- Full audit trail for regulators.
1.3 Scale Estimates
- 10M active users. At market open, 50K orders/sec for the first minutes.
- Quotes: 5,000 stocks with many ticks per second each, fanned out to about 2M connected users.
- 20M orders/day. Each order has ~5 events, so about 100M order events/day.
1.4 API Design
POST /v1/orderswith headerIdempotency-Keyand body{ symbol, side: buy, type: limit, qty: 10, price: 2450.50 }→{ order_id, status: "open" }DELETE /v1/orders/{id}(cancel),PATCH /v1/orders/{id}(modify)GET /v1/orders?status=open,GET /v1/portfolio- WebSocket
/v1/quotes→ subscribe to symbols
High-Level ArchitectureArchitecture
2.1 Overview
- Order Service / OMS (Order Management System): validates orders and tracks their state.
- Risk & Funds Service: checks margin and places a hold on funds (for buys) or shares (for sells).
- Exchange Gateway: keeps persistent connections to the exchange using the FIX protocol (the standard trading message format), sends orders and receives fills.
- Ledger / Positions: the record of cash and shares per user.
- Market Data Service: receives the exchange price feed and streams quotes to users.
- Kafka: the order events log, used by notifications, portfolio updates and audit.
2.2 Architecture Diagram
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
U["Mobile / Web"] --> API["API Gateway"]
API --> OMS["Order Service - state machine"]
OMS --> RISK["Risk and Funds - holds"]
RISK --> LED[("Ledger and Positions DB")]
OMS --> DB[("Orders DB")]
OMS --> EG["Exchange Gateway - FIX"]
EG <--> EX["Stock Exchange"]
EG -->|"fills, rejects"| OMS
OMS --> K[("Order events - Kafka")]
K --> NOTIF["Notifications"]
K --> AUD["Audit store"]
EX -->|"price feed"| MD["Market Data Service"]
MD -->|"WebSocket quotes"| UData ModelData model
CREATE TABLE orders (
order_id UUID PRIMARY KEY,
user_id UUID,
symbol TEXT,
side TEXT, -- buy / sell
type TEXT, -- market / limit / stop
qty INT,
filled_qty INT DEFAULT 0,
limit_price NUMERIC(12,2),
status TEXT, -- pending, open, partially_filled, filled, cancelled, rejected
exchange_order_id TEXT,
version INT
);
CREATE TABLE ledger_entries ( -- append-only money and share movements
entry_id UUID, user_id UUID, asset TEXT, -- 'INR' or a symbol
amount NUMERIC, kind TEXT, -- hold, release, trade, fee
order_id UUID, created_at TIMESTAMP
);
CREATE TABLE fills (exec_id TEXT PRIMARY KEY, order_id UUID, qty INT, price NUMERIC, ts TIMESTAMP);fills.exec_id is the exchange's unique execution ID. Using it as the key means a repeated fill message is ignored.
Key FlowsFlows
4.1 Placing a buy order
- The API checks the idempotency key, so a double tap does not create two orders.
- The Risk service checks available funds and places a hold of
qty × limit price(plus fees). The money is now reserved. - The OMS saves the order as
pendingand sends it to the exchange through the gateway. - The exchange acknowledges →
open. - Fills arrive (maybe several partial fills). For each new
exec_id: updatefilled_qty, turn the matching part of the hold into a real debit, add shares to positions, and publish an event. - When fully filled →
filled, and release any unused hold.
4.2 Cancel
Send a cancel to the exchange. The order is only cancelled when the exchange confirms, because a fill might arrive first. The rest of the hold is then released.
Deep Dive A — Never losing track of an order or a rupeeDeep dive
The exchange sends fills over a connection that drops and replays. A duplicate fill, applied twice, credits shares that do not exist.
Weak
Update the order and the balance in place
A fill arrives: add the quantity to the order, adjust the cash balance, save.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
sequenceDiagram
participant E as Exchange
participant B as Broker
participant DB as Orders and balances
E->>B: fill exec_id 55 - 100 shares at 250
B->>DB: filled_qty += 100, cash -= 25,000
Note over E,B: connection drops, exchange replays
E->>B: fill exec_id 55 - again
B->>DB: filled_qty += 100, cash -= 25,000
Note over DB: 200 shares bought, 50,000 gone, one real tradeThe write carries no identity, so a replay is indistinguishable from a second trade. The balance is a single mutable number with no record of how it got there, so the error is found days later by reconciliation and cannot be traced back.
Good
A state machine with optimistic locking
Give the order explicit states and legal transitions — open → partially_filled → filled, and never backwards — with a version column so two concurrent updates cannot silently overwrite each other.
This fixes concurrent writers and nonsensical transitions, and it is worth having. It does not fix the duplicate: exec 55 applied twice is two legal partially_filled transitions, each one perfectly valid on its own. And the balance is still a number that was overwritten, so there is still no history to audit.
Best
Identity at every boundary, and a ledger instead of a balance
- Idempotency at each hop. The client sends an idempotency key on order creation; we send our own order id to the exchange as the client order id; every fill is keyed by its
exec_id. Applying exec 55 twice is a no-op because the second one is recognised, not because we were lucky with ordering. - An append-only ledger. The balance is not stored and mutated — it is the sum of entries. A buy writes a hold, a fill writes the debit and the share credit, a cancel writes a release. Nothing is edited, so every number can be explained by the rows that produced it, which is also what regulators ask for.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
F["Fill exec_id 55"] --> SEEN{"exec_id already applied?"}
SEEN -->|"yes"| IGN["Ignore - idempotent"]
SEEN -->|"no"| ENT["Append ledger entries - cash debit, shares credit"]
ENT --> ST["Order state transition, version checked"]
ENT --> BAL["Balance = sum of entries"]
ENT --> EOD["End-of-day reconciliation"]
EOD --> CMP["Compare with exchange trade file and clearing house"]Then reconcile every evening against the exchange's trade file and the clearing house. Idempotency and the ledger make mismatches rare; reconciliation is how you find out on the same day when one happens anyway.
Deep Dive B — Market open and quotesDeep dive
- Spike at 9:15 AM: pre-scale before open, accept orders into a queue per exchange connection, and apply per-user rate limits. Orders placed before open ("AMO", after-market orders) are sent in a controlled stream.
- Quotes to millions: the market data service receives the exchange feed once, then fans it out through many WebSocket servers. Users subscribe to specific symbols, and we send throttled updates (e.g., at most 4 per second per symbol per user), because the human eye can't use more.
- In-memory OMS (LLD variant): keep open orders in memory, indexed by order ID and by symbol. Write every change to a log first so the OMS can recover after a crash by replaying it.
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Funds check | Hold before sending the order | Can't overspend | Check after fill: risky |
| Order store | SQL with strict state machine | Correctness, audit | NoSQL: scales easily, weaker transactions |
| Fill handling | Dedupe by exec_id | Safe against repeats | Trust the exchange never repeats: unsafe |
| Quotes | Throttled WebSocket fan-out | Scales to millions | Push every tick: wastes bandwidth |
Common Follow-up QuestionsFollow-ups
- "What if the gateway loses connection mid-order?" The FIX protocol has sequence numbers. On reconnect, both sides replay missed messages. Until confirmed, the order stays
pending. - "How are stop-loss orders handled?" Either the exchange supports them natively, or our service watches prices and places a market order when the trigger price is hit.
- "Portfolio P&L?" Positions × latest price. Compute it on the client from streamed prices to avoid heavy server work.
Wrap-UpWrap-up
Validate, hold funds, then send orders to the exchange through a FIX gateway. Track each order with a strict state machine, apply fills idempotently by execution ID to an append-only ledger, and reconcile with the exchange every day. Handle market-open spikes with pre-scaling and queues, and stream throttled quotes over WebSockets.