Problem RestatementProblem
Design an online bookstore. Users search and browse books, see prices, add books to a cart, place orders, pay, and track them. Databricks asked several versions:
- The full bookstore (browse, search, inventory, orders, payments, order tracking).
- A pricing API that returns prices for one book or a batch of books, and says how fresh each price is.
- A book price aggregator that collects prices from several external sellers, which may be slow, fail, or disagree.
RequirementsRequirements
1.1 Functional
- Search and browse books (title, author, ISBN, category).
getPrice(isbn)andgetPrices([isbn...])with the best price and seller.- Cart, checkout, payment, order status.
- Inventory for books we sell ourselves.
1.2 Non-Functional
- Search and price reads are fast (under 200 ms) even when sellers are slow.
- Prices shown must say how fresh they are, and checkout must use a confirmed price.
- No overselling our own stock, and no double orders.
1.3 Scale Estimates
- 20M books, 10 external sellers.
- 10K price lookups/sec (most are part of list or search pages → batch calls).
- 200 orders/sec at peak.
1.4 API Design
GET
/v1/search?q=&cursor=GET
/v1/prices/{isbn}→ { isbn, best: { seller, price_cents, currency }, as_of, stale: false }POST
/v1/prices:batchGet{ isbns: [...up to 100] } → { results: [...], missing: [...] }POST
/v1/orders(Idempotency-Key) { items: [{ isbn, seller, qty, quoted_price }] }High-Level ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
U["Users"] --> GW["API Gateway"]
GW --> SRCH["Search Service"]
SRCH --> IDX[("Search index")]
GW --> PR["Pricing Service"]
PR --> PC[("Price cache - with as_of")]
PR --> AGG["Seller Aggregator"]
AGG --> S1["Seller A API"]
AGG --> S2["Seller B API"]
AGG --> S3["Seller C API"]
GW --> ORD["Order Service"]
ORD --> INV[("Inventory DB")]
ORD --> PAY["Payments"]
ORD --> ODB[("Orders DB")]- Catalog + Search: book metadata in a DB, indexed for text search.
- Pricing Service: serves prices from a cache (each entry has
as_of), refreshes in the background, and calls the aggregator on misses. - Seller Aggregator: fans out to external seller APIs with timeouts, and normalizes currency and format.
- Order Service: validates, reserves inventory or confirms the price with the seller, and takes payment.
Pricing and Aggregation (the core)
3.1 Batch lookup flow
- Receive up to 100 ISBNs.
- Read them all from the cache in one call. Fresh entries (e.g., under 10 minutes old) are returned directly.
- For missing or stale ones, call the aggregator in parallel with a deadline (e.g., 300 ms total).
- Return what we have by the deadline: fresh prices, stale prices marked
stale: truewith theiras_of, and amissinglist. Never block the whole page on one slow seller. - Late seller answers still update the cache for the next request.
3.2 Talking to external sellers
- Per-seller timeouts and circuit breakers: if Seller B is failing, stop calling it for a while and use cached values.
- Rate limits: respect each seller's quota. Batch ISBNs per seller where their API allows it.
- Background refresh: popular books are refreshed proactively (e.g., every 5 minutes), and rare books only on demand.
- Choosing the best price: lowest total (price + shipping) among in-stock offers, with a deterministic tie-break (e.g., seller rating, then seller ID).
Deep Dive — A price API that depends on sellers you do not controlDeep dive
The batch endpoint takes 100 ISBNs and has to answer in a few hundred milliseconds. The prices live behind third-party seller APIs that are sometimes slow and sometimes down.
Weak
Call every seller and wait
For each ISBN, call the sellers, collect the offers, return the best price.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
REQ["Batch of 100 ISBNs"] --> S1["Seller A - 40 ms"]
REQ --> S2["Seller B - 60 ms"]
REQ --> S3["Seller C - timing out, 30 s"]
S3 --> WAIT["Whole response blocked on the slowest seller"]
WAIT --> PAGE["Product page shows a spinner, then an error"]The response is as slow as the worst seller, every time. One degraded partner takes down the entire catalogue's pricing, and because callers retry, the load on the struggling seller goes up exactly when it is least able to take it.
Good
Put a timeout on each call
Give every seller call a deadline of, say, 300 ms and drop the ones that miss it. The page now renders, and one bad partner costs a little latency rather than the whole request.
It is better and still wasteful: a seller that has been timing out for twenty minutes gets called again on every single request, burning 300 ms of the budget each time to learn what we already knew. And an ISBN whose sellers all missed the deadline returns nothing, so the page has a blank where a price used to be — even though we had a perfectly good price from four minutes ago.
Best
Serve from cache, refresh around it, and be honest about age
Turn the freshness question into part of the contract instead of a hidden failure:
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
REQ["Batch of 100 ISBNs"] --> C[("Cache - one multi-get")]
C --> FRESH["Fresh, under 10 min - return now"]
C --> STALE["Stale or missing"]
STALE --> CB{"Seller circuit open?"}
CB -->|"open"| USE["Use the cached value - mark stale, include as_of"]
CB -->|"closed"| CALL["Call in parallel, 300 ms deadline"]
CALL -->|"answered"| FRESH
CALL -->|"missed the deadline"| USE
CALL -->|"arrives late"| WARM["Still written to cache for the next request"]
BG["Background refresh - popular books every 5 min"] --> CFour things together:
- One multi-get against the cache for all 100 ISBNs, not 100 round trips.
- Circuit breakers per seller. After repeated failures, stop calling that seller for a while and serve cached values. This protects our latency budget and stops us from hammering a partner that is already struggling.
- Return stale data, labelled. Every price carries
as_of, and stale ones are flaggedstale: truewith amissinglist for what we genuinely have nothing for. A four-minute-old price with a timestamp is far more useful to a caller than an empty field, and it lets them decide. - Refresh in the background. Popular books are refreshed on a schedule so their cache entries are almost never stale; the long tail is fetched on demand.
Pick the winning offer deterministically — lowest price plus shipping among in-stock offers, tie-broken by seller rating then seller id — so the same inputs always produce the same answer and the displayed price does not flicker between refreshes.
Orders and Consistency
- The client sends the quoted price and an idempotency key.
- Our own stock: atomically decrement inventory (
UPDATE ... SET qty = qty - n WHERE qty >= n). If 0 rows are updated, it's out of stock. - External seller: confirm the price and availability with the seller's order API. If the price changed, ask the user to confirm.
- Take payment. If payment fails, release the stock or cancel the seller order. This is a small saga: a sequence of steps, each with an undo action.
- Order status updates (shipped, delivered) come from warehouse or seller webhooks and are pushed to the user.
Data ModelData model
books: isbn, title, authors, category, description, cover_url
offers: isbn, seller_id, price_cents, currency, in_stock, as_of (cache + history)
inventory: isbn, warehouse_id, qty
orders: order_id, user_id, status, items (JSON), total_cents, idempotency_key, created_atTrade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Price reads | Cache with as_of + deadline fan-out | Fast pages, honest freshness | Always call sellers live: slow and fragile |
| Batch API | batchGet up to 100 with partial results | One call per page | One call per book: many round trips |
| Seller failures | Timeouts + circuit breakers + stale fallback | Page still works | Fail the whole request |
| Orders | Re-confirm price, saga with undo steps | Correct money and stock | Trust cached price: disputes |
Common Follow-up QuestionsFollow-ups
- "How do you make the price API's freshness explicit?" Return
as_offor every price and astaleflag based on a per-seller freshness target. The client can show "price as of 5 min ago". - "Currency?" Store the seller's currency, convert using a daily FX rate for display, and charge in the user's currency with the rate locked at order time.
- "Search ranking?" Text relevance plus popularity and availability.
Wrap-UpWrap-up
Split the bookstore into catalog and search, pricing, orders, inventory and payments. Serve prices from a cache that records as_of, and on misses fan out to external sellers in parallel with per-seller timeouts, circuit breakers and a global deadline, returning partial or stale results with clear freshness flags. At checkout, re-confirm the price, decrement stock atomically, and use a saga with idempotency so every order is created once and cleaned up correctly on failure.