Problem RestatementProblem
Design an online chess service. Players get matched with an opponent of similar skill, play a real-time game with a chess clock (e.g., 5 minutes each), and see each other's moves instantly. Games end by checkmate, resignation, draw or timeout (running out of clock time). If a player's connection drops, they should be able to reconnect and continue. Others can watch as spectators, and finished games are saved.
A nice way to answer (asked at OpenAI): start with a simple design for launch, then show how it changes as traffic grows.
RequirementsRequirements
1.1 Functional
- Matchmaking by rating and time control (rated and unrated).
- Real-time moves with legal-move validation.
- A clock per player that the server controls.
- Reconnect to an ongoing game.
- Spectators, game history and rating updates after each game.
1.2 Non-Functional
- Low latency: moves appear for the opponent in under ~100–200 ms.
- Fairness: nobody gains time from network lag or tricks, and the server is the referee.
- Reliability: a server crash should not lose the game.
1.3 Scale Estimates
- 10M daily users, 500K games in progress at peak.
- A move every ~10 seconds per game → 50K moves/sec.
- Each move is tiny (about 100 bytes). Bandwidth is small; the challenge is lots of long-lived connections.
- Each server holds ~50K WebSocket connections, so we need tens of game servers.
1.4 API Design
POST /v1/matchmaking{ time_control: "5+0", rated: true }, then wait formatch_foundon the socket.- WebSocket
/v1/games/{game_id}: - client → server:
{ type: "move", move: "e2e4", seq: 17 } - server → clients:
{ type: "moved", move: "e2e4", fen, clocks: { white_ms, black_ms }, seq: 17 } GET /v1/games/{id}(history, PGN)
High-Level ArchitectureArchitecture
2.1 Overview
- Matchmaking Service: queues players per time control and rating band and pairs them.
- Game Servers: each game lives on one game server that holds the board and clocks in memory. Both players (and spectators) connect to that server.
- Game Router: maps
game_id → game server, e.g., a Redis lookup or consistent hashing. - Game Store: saves moves (so a crashed game can be rebuilt) and finished games.
- Rating Service: updates Elo/Glicko ratings when a game ends.
- Spectator fan-out: pub/sub, so many watchers don't overload the game server.
2.2 Architecture Diagram
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
P1["Player White"] -->|"WebSocket"| GS["Game Server - owns game in memory"]
P2["Player Black"] -->|"WebSocket"| GS
P1 --> MM["Matchmaking Service"]
P2 --> MM
MM -->|"create game"| GR["Game Router"]
GR --> GS
GS -->|"append each move"| LOG[("Move log - Redis/Kafka")]
GS -->|"game over"| DB[("Game history DB")]
GS --> PS[("Pub/Sub")]
PS --> SP["Spectators"]
DB --> RT["Rating Service"]Data ModelData model
games: game_id, white_id, black_id, time_control, status (active/finished),
result, started_at, ended_at
moves: game_id, seq, move (UCI "e2e4"), white_ms_left, black_ms_left, server_ts
ratings: user_id, time_control, rating, deviation, games_playedKey FlowsFlows
4.1 Matchmaking
- Players join a queue for their time control, stored per rating bucket (e.g., 1400–1500).
- Every second, the matchmaker pairs players in the same bucket. If someone waits too long, it widens the allowed rating gap gradually.
- It creates the game, picks a game server, and tells both players where to connect.
4.2 Making a move
- The client sends
{ move, seq }. - The game server checks it is that player's turn, the move is legal, and
seqis the expected next number (which drops duplicates). - It updates the board and the clocks: the mover's clock stops, and the opponent's starts, using server time.
- It appends the move to the move log, then broadcasts it to both players and spectators.
Deep Dive A — Who owns the clockDeep dive
A blitz game is decided by tenths of a second. Whoever holds the authoritative clock decides who wins on time, so this is a trust question before it is a timing question.
Each client runs its own clock
The browser counts down locally and tells the server "I moved with 43.2 seconds left".
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
sequenceDiagram
participant W as White client
participant S as Server
participant B as Black client
W->>S: move e4 - my clock says 43.2s left
S->>B: relay move
Note over W: patched client reports 43.2s no matter how long it thought
B->>S: my clock says White ran out
Note over S: two clients, two clocks, no way to tell who is rightThe clock is now a number the player controls, and a modified client simply never runs out. Even with honest players the two clocks drift apart, so both sides watch a different game.
The server checks the clock when a move arrives
The server records the time of each move and subtracts the elapsed time from the mover's clock. Clients only display a countdown. Nobody can cheat by editing a number.
One case is still unhandled, and it is the common one: a player who is losing simply stops moving. No move arrives, so nothing triggers the subtraction, and the game hangs until someone reconnects and pokes it. Losing on time is a result the system has to produce on its own.
The server owns the clock and fires the timeout itself
When a player's turn begins, the server starts a timer for exactly their remaining time. A move cancels it and starts the opponent's; if it fires first, the server ends the game on time and broadcasts the result.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
M["Move arrives"] --> C["Subtract elapsed time from mover's clock"]
C --> LAG["Credit measured round-trip, capped at 100 ms"]
LAG --> T["Arm timer for the opponent's remaining time"]
T -->|"move arrives first"| M
T -->|"timer fires"| END["Game over on time - server decides"]Add lag compensation: credit back the measured round-trip time, capped at something like 100 ms, so a player on a slow connection is not taxed for the network. Cap it, or the compensation becomes the exploit.
The rule to state out loud: the clock keeps running while a player is disconnected. That is how over-the-board chess works, and players expect it.
Deep Dive B — Reconnects and crashesDeep dive
- Reconnect: the client reconnects with
game_idand its lastseq. The server sends the full position and clocks, plus any missed moves. The player's clock keeps running while they are disconnected; that is the rule. - Game server crash: because every move was appended to a durable move log before broadcast, the router assigns the game to another server. That server rebuilds the board from the log, and players reconnect automatically. Clock time lost during the switch can be credited back.
- Evolving the design:
- Launch: one server holds everything, with Postgres for games. Simple.
- Growth: many game servers, sticky routing by game, Redis for queues and move logs.
- Large scale: regional clusters (match players in the same region for low latency), a separate spectator fan-out, and async rating and history pipelines.
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Game state | In memory on one owning server | Fast, simple ordering | Stateless servers + shared DB per move: slower, lock contention |
| Transport | WebSockets | Two-way, low latency | Polling: laggy, wasteful |
| Durability | Append move before broadcast | Recover after crash | Save only at game end: lose games on crash |
| Clock | Server-authoritative | Fair, cheat-resistant | Client clocks: easy to cheat |
Common Follow-up QuestionsFollow-ups
- "Undo / takeback?" It needs the opponent's consent. Pop the last move from the in-memory state and append an "undo" event to the log.
- "Anti-cheat?" After games, compare moves with a chess engine. Very high engine-match rates for a player get flagged for review.
- "Leaderboard?" Keep a sorted set of ratings per time control and update it asynchronously when games end.
Wrap-UpWrap-up
Pair players by rating in matchmaking, then give each game one owning game server that holds the board and a server-authoritative clock in memory. Players connect over WebSockets. Validate every move with sequence numbers and append it to a durable log before broadcasting, so reconnects and server crashes can restore the exact game.