Problem RestatementProblem
Design a system that recommends items to users: posts, videos, games, local sports teams or weekly deals. When a user opens the app, we must pick the best 20–50 items out of millions, in about 100–200 ms, and the picks should feel personal, fresh and varied.
The standard answer splits the work into stages. Candidate generation quickly finds a few hundred possibly good items. Ranking then scores those few hundred carefully with a machine learning model.
RequirementsRequirements
1.1 Functional
- Return a ranked list of items for a user (and optional context: location, time, page).
- Learn from user actions: views, clicks, likes, watch time, purchases, skips.
- Handle new users and new items (cold start).
- Avoid showing the same thing again and again. Keep variety.
1.2 Non-Functional
- Latency: under ~200 ms end to end.
- Scale: 100M daily users, peaks of 100K requests/sec.
- Freshness: new items and new user behavior should show up within minutes to hours.
1.3 Scale Estimates
- 50M items in the catalog. Scoring all of them per request is impossible (50M × 100K/sec), which is why candidate generation exists.
- Interaction events: 100M users × 50 actions = 5B events/day feeding training.
1.4 API Design
/v1/recommendations?user_id=42&surface=home&limit=30→ [{ item_id, score, reason }]/v1/events{ user_id, item_id, action, ts }High-Level ArchitectureArchitecture
2.1 Overview
- Event logging: user actions go to Kafka, then to a data lake.
- Offline training (daily): trains models such as a two-tower embedding model for candidates and a ranking model (gradient-boosted trees or a neural net).
- Embedding index: item vectors stored in an approximate nearest neighbor index (FAISS, ScaNN). This finds the items whose vectors are closest to the user's vector very quickly.
- Feature store: precomputed user and item features (e.g., "user's favorite categories", "item click rate last hour") for fast lookup.
- Recommendation service: collects candidates, fetches features, ranks, re-ranks, and returns.
2.2 Architecture Diagram
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
U["User opens app"] --> RS["Recommendation Service"]
RS --> CG1["Candidates: embedding ANN"]
RS --> CG2["Candidates: follows / friends"]
RS --> CG3["Candidates: trending / local"]
RS --> FS[("Feature Store")]
RS --> RK["Ranking model server"]
RK --> RR["Re-rank: diversity, filters"]
RR --> U
U -->|"clicks, views"| K[("Kafka")]
K --> DL[("Data lake")]
DL --> TR["Offline training"]
TR --> CG1
TR --> RK
K --> FSStage 1 — Candidate Generation
Gather ~500–1,000 candidates from several simple sources:
- Collaborative filtering / embeddings: "people who liked what you liked also liked X". A two-tower model turns each user and item into a vector, and similar vectors mean a good match. Find the nearest items with the ANN index.
- Social: items from friends or accounts you follow.
- Content-based: items similar to ones you recently engaged with (same category, same artist).
- Context: trending now, popular near your location (e.g., local sports teams), new arrivals.
Using several sources makes the result robust. If one source is weak for a user, others fill the gap.
Stage 2 — Ranking
- For each candidate, fetch features: user features, item features, and user–item features (e.g., "user clicked this category 12 times this week").
- The ranking model predicts probabilities: will the user click, watch, like? Combine them into one score, e.g.,
score = 0.6 × P(click) + 0.4 × P(watch 30s). - Sort by score.
- Diversity: no more than 2 items in a row from the same creator or category.
- Freshness: give a small boost to new items.
- Filters: remove already-seen, blocked or unavailable items.
Deep Dive A — The user who just signed upDeep dive
Recommendations are built from behaviour, and a brand-new user has none. The first session decides whether they come back, so this is not an edge case.
Show everyone new the global top items
Serve the most popular items in the catalogue until there is enough history to personalise.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
NEW["New user - no history"] --> POP["Global top 50"]
POP --> SAME["Every new user sees the same list"]
SAME --> RICH["Popular items get more clicks"]
RICH --> POP
SAME --> BORE["Nothing matches a niche interest - user leaves"]It is not wrong, it is generic — the same list a competitor shows, with nothing to suggest the product knows anything about this person. It also feeds the popularity loop: the top items get all the new-user traffic, which keeps them on top.
Use what the item is, not who clicked it
For a new item, place it near similar items using its own content: title, category, tags, an image or text embedding. For a new user, use whatever context exists — country, device, language, the "pick three interests" screen at sign-up.
That gets a reasonable first page without any interaction history. But content similarity says what an item resembles, not whether people like it, and a new item placed by its embedding still gets no traffic unless something deliberately gives it some. The cold start never ends on its own.
Context to start, exploration to escape, session signals to converge
Three pieces, each fixing what the previous rung left:
- Context for the first request. Location, device, language and sign-up answers give a personalised-enough first page in place of the global list.
- An exploration budget. Reserve a small slice of every feed — a few percent — for items the model is uncertain about, and log what was shown, not only what was clicked. Without impressions in the training data the model cannot learn that an item was offered and ignored, and the popularity loop closes permanently.
- Session signals in the request. Feed the user's last few actions straight into ranking, so page two already reflects page one. A new user becomes a known user inside a single session rather than overnight.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
CTX["Context - country, device, sign-up interests"] --> R1["First page"]
R1 --> ACT["Clicks, skips, dwell in this session"]
ACT --> R2["Next page - already adapting"]
EXP["Exploration slice - few percent"] --> R1
EXP --> R2
R1 --> LOG[("Log impressions AND clicks")]
R2 --> LOG
LOG --> TRAIN["Training - learns from what was shown too"]Judge all of it with an A/B test on retention, not offline AUC. Offline metrics are scored against logs the current model produced, so they systematically flatter whatever is already running.
Deep Dive B — Freshness and feedback loopsDeep dive
- Near-real-time features: stream processors update counters such as "item clicks in the last hour" in the feature store, so trending items rise quickly.
- Session signals: include the user's last few actions in the request, so recommendations react within the same session.
- Feedback loops: the model only learns from what it showed. Without care, popular items get more popular and new ones never appear. Fix this by showing a small percentage of exploratory items and logging which items were shown, not just clicked.
- Measurement: offline metrics (AUC, recall@K) help, but the real decision comes from A/B tests on engagement, retention and satisfaction.
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Architecture | Candidate gen + ranking | Fast and accurate | Score everything: impossible at scale |
| Candidates | Many sources merged | Robust, explainable | One embedding model: simpler, less coverage |
| Serving | Online ranking per request | Uses fresh context | Precompute lists nightly: cheap, stale |
| Features | Feature store (batch + streaming) | Same features in training and serving | Compute in the service: drifts from training |
Common Follow-up QuestionsFollow-ups
- "How do you keep latency low?" Run candidate sources in parallel, cap candidates at ~500, batch feature lookups, and cache recommendations for a few minutes.
- "How do you explain recommendations?" Keep the candidate source as a reason ("Because you watched X", "Popular near you").
- "How do you avoid training/serving skew?" Compute features once in the feature store and log the exact features used at serving time for training.
Wrap-UpWrap-up
Use two stages: gather a few hundred candidates from embeddings, the social graph, content similarity and trending lists, then rank them with an ML model using features from a feature store. Re-rank for diversity and freshness, solve cold start with content and exploration, and judge changes with A/B tests.