Problem RestatementProblem
Design the product catalog for a large marketplace such as Walmart, Amazon or Pinterest shopping. Merchants add and edit products: titles, images, variants (size and color), stock and prices. Sometimes they upload bulk edits of millions of products at once. Shoppers read product pages and search results at very high traffic. Changes must show up quickly and consistently on the product page, in search and in caches.
A common variant: a price changes at most once a day, but product pages read prices constantly. How do we serve prices with very low latency?
RequirementsRequirements
1.1 Functional
- Create and update products and variants (single and bulk).
- Serve product details and prices for product pages and lists.
- Keep search, recommendations and caches in sync with changes.
- Show price and availability accurately, and validate them again at checkout.
1.2 Non-Functional
- Reads: very high QPS with low latency (under 50 ms).
- Writes: heavy bursts from bulk uploads without hurting read latency.
- Freshness: changes visible within seconds to a few minutes.
- Correctness at checkout: the charged price must be the current real price.
1.3 Scale Estimates
- 500M products (SKUs), ~5 KB each → 2.5 TB of catalog data.
- Reads: 200K product views/sec at peak.
- Writes: 5K updates/sec normally. A bulk edit can bring 10M updates in an hour (~3K/sec extra).
1.4 API Design
/v1/products/{id}and GET /v1/products?ids=1,2,3 (batch for lists)/v1/products/{id}{ price, stock, title, ... } (merchant)/v1/merchants/{id}/bulk-updates(upload CSV/JSON) → { job_id }, then GET /v1/bulk-updates/{job_id}High-Level ArchitectureArchitecture
2.1 Overview
- Catalog Write Service: validates changes and writes them to the source-of-truth DB (sharded by product ID).
- Bulk Import Service: splits huge files into chunks, validates them, and feeds them to the write path at a controlled rate.
- CDC stream: change data capture, which means reading every committed DB change as an event (e.g., Debezium → Kafka).
- Read model: a denormalized "product page view" document in a fast key-value store (e.g., DynamoDB/Redis), built from the CDC stream.
- Search indexer: updates the search index from the same stream.
- Caches + CDN: product pages and images cached at the edge.
2.2 Architecture Diagram
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
M["Merchants"] --> WS["Catalog Write Service"]
M -->|"bulk file"| BI["Bulk Import - chunk, validate, throttle"]
BI --> WS
WS --> DB[("Catalog DB - source of truth")]
DB -->|"CDC"| K[("Kafka - product changes")]
K --> RM["Read-model builder"]
RM --> KV[("Product view store")]
K --> SI["Search indexer"]
K --> INV["Cache invalidator"]
INV --> CDN["CDN / page cache"]
S["Shoppers"] --> CDN
CDN --> RS["Product Read Service"]
RS --> KVData ModelData model
products: product_id, merchant_id, title, description, category_id, brand, attributes (JSON), status
variants: variant_id, product_id, size, color, sku
prices: variant_id, price_cents, currency, valid_from, version
inventory: variant_id, warehouse_id, available_qtyThe read model joins all of these into one document per product. A product page then needs one fast lookup instead of five joins.
Key FlowsFlows
4.1 A merchant changes a price
- The write service updates
prices(bumpingversion) in the source DB. - CDC emits the change. The read-model builder updates the product document, and the invalidator purges the CDN or page cache for that product.
- The search indexer updates the price field so price filters stay correct.
- The new price is visible within seconds.
4.2 Bulk edit of 10M products
- The merchant uploads a file and gets a job ID.
- The import service splits it into chunks of 1,000 rows, validates each row, and records bad rows in an error report.
- It writes chunks at a throttled rate (e.g., max 2K rows/sec per merchant), so normal traffic isn't affected.
- Each chunk is idempotent (keyed by
job_id + chunk_no), so retries don't double-apply. - The merchant sees progress and a downloadable error report.
Deep Dive A — One catalogue, two completely different workloadsDeep dive
Merchants write products; shoppers read them. The read side is thousands of times larger and wants different data in a different shape.
One database for both
Merchants write to a normalised schema — products, variants, images, prices, inventory — and product pages read from it.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
M["Merchant bulk edit - 2M products"] --> DB[("Normalised catalogue DB")]
S["Shoppers - 200K page views/sec"] --> DB
DB --> J["Each page = 6 joins across variants, images, price, stock"]
J --> SLOW["Bulk edit saturates the DB - every product page slows"]A product page needs half a dozen joins, and a bulk edit of two million rows lands on the same machine. The shopping experience degrades whenever a merchant does something entirely normal, and neither workload can be tuned without hurting the other.
Read replicas
Send reads to followers and keep writes on the leader. Read capacity scales out and bulk edits stop blocking page loads.
The contention is gone, but the shape problem is not: replicas run the same six joins, just on more machines. Scaling reads now means scaling join work, and caching is awkward because there is no single row that represents "the product page" to invalidate.
Build a read view shaped like the page
Keep the normalised database as the source of truth for writes, and project changes into a denormalised read view — one document per product, holding everything the page renders.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
W["Merchant writes"] --> SRC[("Normalised DB - source of truth")]
SRC --> EV["Change events"]
EV --> PROJ["Projector"]
PROJ --> VIEW[("Read view - one document per product")]
VIEW --> CDN["Cache / CDN"]
CDN --> SHOP["Shoppers"]
CO["Checkout"] -->|"re-read price and stock"| SRCA page is now one key lookup, trivially cacheable and trivially invalidated. The view is eventually consistent — a second or two behind — and for browsing that is invisible and entirely acceptable.
The line to draw clearly, because the interviewer will push on it: checkout does not read the view. The order service re-reads price and stock from the source of truth, and if the price has changed the buyer is told before paying. Browsing tolerates staleness; taking money does not. Saying which reads may be stale, and which may never be, is the whole point of splitting the paths.
Deep Dive B — Prices read constantly, changed dailyDeep dive
- Prices change at most once a day, so cache aggressively: an in-memory cache on each read server plus the CDN, with a long TTL (e.g., 1 hour), and explicit invalidation on change so we're never stale for long.
- For product lists, use batch reads (
ids=1..50) to avoid 50 separate calls. - If price changes are scheduled ("new price at midnight"), precompute and push them to caches just before they take effect.
- Hot products (a flash sale): the CDN absorbs traffic, and request coalescing on cache misses stops thousands of requests from hitting the DB at once.
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Read/write split | Source DB + denormalized read model | Fast reads, safe writes | One DB with joins: simple, slow at scale |
| Syncing | CDC → Kafka | Every consumer sees every change in order | Dual writes from the service: can miss updates |
| Bulk edits | Chunked, throttled, idempotent jobs | Doesn't hurt shoppers, safe retries | Direct bulk writes: can overload the DB |
| Freshness | Eventual for browsing, strict at checkout | Best of both | Strict everywhere: expensive |
Common Follow-up QuestionsFollow-ups
- "Two updates to the same product arrive out of order?" Use a
versionnumber and ignore older versions in the read-model builder. - "How do you keep search in sync?" The same CDC stream feeds the indexer, and a nightly job compares counts or checksums to catch drift.
- "Many merchants selling the same product?" Keep a canonical product and a separate
offerstable for each merchant's price and stock, and pick the "buy box" winner in the read model.
Wrap-UpWrap-up
Write product changes to a sharded source-of-truth DB, stream every change with CDC into Kafka, and build a denormalized product view plus search index from that stream. Serve reads from the view, caches and the CDN with invalidation on change. Handle bulk edits as chunked, throttled, idempotent jobs, and always re-check price and stock at checkout.