Problem RestatementProblem
Design a service like CamelCamelCamel (asked at Meta twice). Users paste a product link from a large online store, see the product's price history chart, and set an alert: "tell me when this drops below $250". The system periodically fetches current prices for millions of products (through official APIs or scraping, within rate limits), stores the history, and notifies users when a price crosses their threshold.
RequirementsRequirements
- Add a product by URL (normalize it to a product ID, e.g., an ASIN).
- Price history chart (daily or hourly points).
- Alerts: below a price, or X% drop. One-time or recurring.
- Notifications by email or push, without spam.
1.1 Scale Estimates
- 100M tracked products, 20M active alerts.
- If we fetched every product hourly: 100M/hour ≈ 28K fetches/sec, likely beyond the source's limits. So we prioritize.
- History: 100M products × 1 point/day × 16 bytes ≈ 1.6 GB/day, which is fine.
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
U["Users"] --> API["API - products, alerts, history"]
API --> PDB[("Products + alerts DB")]
SCH["Fetch Scheduler - priority"] --> FQ[("Fetch queue")]
FQ --> FE["Fetchers - API/scraper, rate limited"]
FE --> STORE["Marketplace"]
FE --> K[("Price updates")]
K --> HIST[("Price history - time series")]
K --> AM["Alert Matcher"]
AM --> PDB
AM --> N["Notifier - dedupe"]
N --> UData ModelData model
products: product_id, marketplace, url, title, last_price, last_fetched_at, next_fetch_at, priority
price_points: product_id, ts, price_cents, availability (time-series, partitioned by month)
alerts: alert_id, user_id, product_id, type (below|pct_drop), threshold_cents, active, last_notified_priceIndex alerts by (product_id, threshold_cents), so when a product's price changes, we can quickly find alerts where threshold >= new_price.
Key FlowsFlows
4.1 Fetch scheduling
- Each product has a
next_fetch_atand a priority: - Popular products (many alerts or views) → every hour.
- Volatile products (prices change often) → more often.
- Products with no alerts and few views → daily or weekly.
- The scheduler pulls due products (indexed by
next_fetch_at) into the queue. - Rate limits: fetchers use a token bucket per marketplace (and per API key), prefer official APIs with batch lookups (e.g., 10 products per call), and fall back to scraping carefully (respect robots.txt, back off on errors).
4.2 Price update
- The fetcher gets a price and publishes
{ product_id, price, ts }if it changed (or every day as a heartbeat point). - History stores the point.
products.last_priceis updated. - The Alert Matcher runs only when the price dropped:
SELECT alerts WHERE product_id = ? AND threshold_cents >= new_price AND active. - For each match, notify, unless we already notified this user at this or a lower price (
last_notified_price). One-time alerts are deactivated.
Deep Dive — Refreshing millions of prices inside a rate limitDeep dive
Ten million tracked products, an upstream API that allows a few hundred requests a second. The schedule is the design.
Poll everything on a fixed cycle
Every tracked product is checked every hour, in a loop.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
P["10,000,000 products"] --> H["Every hour"]
H --> RATE["2,800 requests/sec needed"]
RATE --> LIM["Upstream allows ~300/sec"]
LIM --> T["429s, then a block"]
H --> WASTE["Most products' prices did not change"]The arithmetic does not fit, and nearly all of the budget is spent confirming that prices have not moved. When the limit is hit the upstream starts rejecting — and a blocked API key means no product gets checked, including the ones that mattered.
Only poll what someone is watching
Skip products with no active alerts and no recent chart views. On a real catalogue this removes the large majority of the work.
The fit is better and the policy is still flat: a product with one watcher and a stable price for six months is checked exactly as often as a graphics card with 40,000 watchers that moves twice a day. The budget is spread evenly across items that deserve wildly different attention.
Spend the budget where it pays
Give every product a score from how much people care and how much it moves, and let the score drive its cadence:
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
W["Watcher count"] --> SC["Priority score"]
V["Recent price volatility"] --> SC
N["Proximity to someone's threshold"] --> SC
SC --> T1["Hot - every 10 minutes"]
SC --> T2["Normal - hourly"]
SC --> T3["Cold - daily"]
T1 --> TB["Token bucket per source"]
T2 --> TB
T3 --> TB
TB --> F["Fetcher"]
F -->|"429 or 5xx"| BO["Exponential backoff, tier demoted"]
F -->|"price changed"| EV["Price event - history + alert matching"]- Proximity to a threshold is the signal people forget and the most valuable one here. A product sitting $5 above someone's alert deserves far more attention than one $400 above it — that is where a poll can actually produce a notification.
- A token bucket per source enforces the upstream limit centrally, so no amount of tier misconfiguration can exceed it. Priority decides who spends the tokens; the bucket decides how many exist.
- Back off on rejection, and demote. A 429 slows that source and pushes its products down a tier, rather than retrying into a block that would stop everything.
Store the history as change events, not samples — a row only when the price actually moves. Ten million products sampled hourly is a billion rows a day of mostly identical numbers; the same information as change events is a tiny fraction of that, and the chart is drawn by holding each value until the next change.
Details Worth Mentioning
- Validation: ignore obviously wrong prices (parser errors, a $0 price, a third-party seller at a weird price). Confirm a big drop with a quick re-fetch before alerting.
- Deal spikes: a big sale may trigger 1M alerts at once. The notifier queues and rate-limits sending (and batches emails).
- URL normalization: many URLs map to the same product (tracking params, mobile vs desktop), so extract the canonical product ID.
- History charts: downsample old data (daily min, max and close), and cache chart images or JSON for popular products.
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Fetch frequency | Priority by popularity and volatility | Freshness where it matters, within limits | Fetch everything hourly: exceeds limits |
| Data source | Official API first, scraping fallback | More reliable, allowed | Scraping only: fragile, blocking risk |
| Alert matching | Index by (product, threshold), only on drops | Few rows checked | Scan all alerts on each update: slow |
| Notifications | Dedupe by last notified price | No spam | Notify on every fetch below threshold: spam |
Wrap-UpWrap-up
Normalize product URLs to canonical IDs and schedule price fetches by priority (popularity and volatility) with per-marketplace rate limits, preferring batched official APIs. Stream price changes into a time-series history and an alert matcher that, on price drops, finds alerts via a (product, threshold) index and sends deduplicated, rate-limited notifications. Validate suspicious prices before alerting, and downsample history for cheap long-term charts.