Problem RestatementProblem
Design a product like ChatGPT. A user types a message and the answer streams back word by word. Conversations are saved, so the user can come back later on any device and continue. The answers come from large language models (LLMs) running on GPUs, which are expensive and limited, so the system has to share them fairly and stay up even when demand is higher than capacity.
A good way to answer: start with a minimal version (no history, just question in and answer out), then add conversations, streaming, limits and reliability.
RequirementsRequirements
1.1 Functional
- Send a message and receive a streamed response.
- Keep conversations: list, open, rename, delete.
- Continue a conversation, which means the model sees the earlier messages.
- Stop generating, and regenerate an answer.
- Different plans (free vs paid) get different models and limits.
1.2 Non-Functional
- Time to first token under ~1 second in normal load.
- High availability: degrade gracefully (smaller model, queue, clear message) instead of failing.
- Scale: tens of millions of daily users.
- Safety: filter harmful inputs and outputs.
1.3 Scale Estimates
- 50M daily users × 10 messages = 500M messages/day ≈ 6,000/sec, peak ~20K/sec.
- An average answer is 400 tokens. At ~50 tokens/sec per stream, each answer takes about 8 seconds, so ~50K–150K streams are open at the same time.
- The GPU fleet is the bottleneck. Each GPU server handles only a limited number of concurrent requests (e.g., 50–100 with batching), so we need thousands of GPU servers.
- Storage: 500M messages × 2 KB ≈ 1 TB/day of conversation text.
1.4 API Design
/v1/conversations/{id}/messageswith { content, model? } → an SSE stream (Server-Sent Events: one long HTTP response that sends small chunks as they are ready) of { delta: "Hello" } … { done: true, message_id }./v1/conversations?cursor=and GET /v1/conversations/{id}./v1/messages/{id}/stop.High-Level ArchitectureArchitecture
2.1 Overview
- API Gateway: login, rate limits per user and plan, and SSE connections.
- Chat Service: loads conversation history, builds the prompt (context), applies safety checks, and calls the inference layer.
- Inference Router: picks a model cluster with free capacity and handles queueing.
- Model servers (GPU): run the LLM, batching many requests together to use the GPU well.
- Conversation Store: messages by conversation (e.g., Cassandra/DynamoDB), plus a cache for recent conversations.
- Safety service: checks prompts and outputs.
2.2 Architecture Diagram
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
U["Web / Mobile"] -->|"SSE"| GW["API Gateway - auth, rate limits"]
GW --> CS["Chat Service"]
CS --> DB[("Conversation Store")]
CS --> SF["Safety Service"]
CS --> R["Inference Router"]
R --> Q[("Priority queues per model")]
Q --> M1["GPU cluster - large model"]
Q --> M2["GPU cluster - small model"]
M1 -->|"tokens"| CS
M2 -->|"tokens"| CSData ModelData model
conversations: conversation_id, user_id, title, created_at, updated_at, model
messages: conversation_id (partition), message_id (time-ordered), role (user/assistant),
content, token_count, parent_message_id (for regenerate branches), statusPartitioning by conversation_id keeps a whole conversation together, so loading it is one fast query.
Key FlowsFlows
4.1 Sending a message
- The gateway checks the user's rate limit (e.g., 40 messages / 3 hours on the free plan).
- The Chat Service saves the user message, then loads recent messages.
- Context assembly: models have a maximum input size (the context window). If the conversation is too long, keep the system prompt plus the newest messages, and replace the old part with a short summary.
- A safety check runs on the input.
- The router sends the request to a GPU cluster. Tokens stream back and are forwarded to the user through SSE as they arrive.
- When the stream ends, save the full assistant message (and generate a conversation title in the background for new chats).
4.2 Stop and disconnect
If the user clicks stop or closes the tab, the Chat Service cancels the GPU request, which frees capacity, and saves the partial answer.
Deep Dive A — Getting work out of expensive GPUsDeep dive
A GPU is the most expensive thing in this system by a wide margin, so how requests are fed to it decides the unit economics.
One request per GPU at a time
The router sends a request to a free GPU, waits for the answer, then sends the next one. Simple, and the latency for that one user is as good as it gets.
Generating a token is a matrix multiply barely large enough to keep the hardware busy. Serving one conversation leaves most of the GPU idle between tokens — throughput lands at a small fraction of what the card can do, and the bill is the same either way. At any real traffic level the queue grows without the hardware ever working hard.
Fill a batch, then run it
Collect requests until you have, say, 32 of them, and run them through the model together. The same matrix multiply now serves 32 users, and throughput jumps by an order of magnitude.
Two costs appear. Request one waits for request 32 to show up before anything starts, so quiet periods add latency for no reason. And the batch runs until its slowest member finishes — one user asking for a 2,000-token essay holds 31 short answers hostage, because a slot cannot be released until the whole batch is done.
Continuous batching, with a queue that pushes back
Let requests join and leave the running batch independently. As soon as one sequence emits its final token, its slot is freed and the next waiting request takes it — no waiting for the batch to fill, no waiting for the slowest member.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
Q["Request queue - paid tier first, free tier guaranteed share"] --> R["Router"]
R --> B["Running batch on the GPU"]
B -->|"sequence finishes - slot freed"| R
R -->|"queue too long"| REJ["Reject early - at capacity"]
R -->|"big model saturated"| SM["Smaller, cheaper model"]Around it, three policies that matter as much as the batching:
- Backpressure. When the queue is longer than the wait budget, reject immediately with a clear "at capacity" message. Letting everyone in and timing out is strictly worse — the user waits, and the GPU work is thrown away.
- Priority with a floor. Paid requests are served first, but free users keep a guaranteed share so they are never starved out entirely.
- Degrade instead of failing. When the large model is saturated, route free traffic to a smaller one. A quicker, slightly weaker answer beats an error.
Autoscaling is not one of the answers here: a GPU node takes minutes to boot and load weights, so capacity has to be planned ahead of the daily peak, not summoned during it.
Deep Dive B — Reliability of long streamsDeep dive
- A stream can take 30+ seconds, and many things can break in the middle. Save tokens in chunks as they arrive, so a page refresh can resume by reading what has been generated so far.
- If a GPU server dies mid-answer, the router retries on another server (starting over) and the UI shows a "regenerating" state.
- Make message sends idempotent with a client-generated message ID, so a network retry does not create two answers.
- For multi-region, users are served in their nearest region. Conversation data is replicated so a region failure only interrupts in-flight streams.
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Streaming | SSE | Simple, works over HTTP, one direction is enough | WebSockets: two-way, more complex |
| Long context | Recent messages + summary | Fits the context window, keeps cost down | Full history: better recall, expensive, may not fit |
| Overload | Queue + priority + smaller-model fallback | Stays up under peaks | Hard reject: simpler, worse experience |
| Storage | Wide-column DB by conversation | Fast reads, scales | SQL: easy at first, harder to shard |
Common Follow-up QuestionsFollow-ups
- "How do you add memory across conversations?" Store extracted user facts separately and retrieve the relevant ones into the prompt, the same way retrieval-augmented generation (RAG) works.
- "How do you count cost?" Count input and output tokens per request, add them to per-user and per-org usage counters, and use those for limits and billing.
- "Enterprise version?" Add SSO, per-organization data isolation, audit logs, and a setting to disable training on customer data.
Wrap-UpWrap-up
Stream answers over SSE from a Chat Service that loads history, builds the context and calls an inference router. GPUs are the scarce resource, so use continuous batching, priority queues, early rejection and smaller-model fallback. Store conversations partitioned by conversation ID, save streamed tokens as they arrive, and make sends idempotent so failures mid-answer are recoverable.