Problem RestatementProblem
Design a system that records what users do in web and mobile apps: page_view, click, app_install, add_to_cart, purchase. Product teams across several products use it to answer questions like "how many daily active users do we have?", "where do users drop off between signup and purchase?" (a funnel), or simply "how many people viewed this product page?"
Asked many times at Rippling, and at Uber as "product page view tracking".
RequirementsRequirements
1.1 Functional
- A client SDK to send events with properties (user, session, device, page, product ID).
- Collect events from many products with a shared schema.
- Real-time counts (e.g., views of a product page in the last hour) and historical analysis (DAU, retention, funnels).
- Dashboards and ad-hoc queries.
1.2 Non-Functional
- High ingest throughput and no slowdown for the app itself.
- Low data loss: a small loss (under 0.1%) is acceptable for analytics, but not large gaps.
- Freshness: real-time panels within ~1 minute, and batch reports within hours.
- Privacy compliance: consent, and deleting a user's data on request (GDPR).
1.3 Scale Estimates
- 50M daily users × 100 events = 5B events/day ≈ 60K/sec, peak ~200K/sec.
- ~500 bytes each → 2.5 TB/day raw, less after columnar compression (~5–10x).
1.4 API Design
- SDK:
track("add_to_cart", { product_id: 991, price: 1299 })→ batched. - Collector:
POST /v1/eventswith[{ event_id, event_name, user_id, anonymous_id, session_id, ts, properties }] - Query:
GET /v1/metrics/page-views?product_id=991&window=1hand SQL access for analysts.
High-Level ArchitectureArchitecture
2.1 Overview
- Client SDK: batches events (every 10 seconds or 50 events), stores them locally when offline, retries with backoff, and attaches a unique
event_id. - Collector: a thin, stateless HTTP service that validates, adds server time and IP-based geo, and writes to Kafka.
- Schema registry: defines allowed events and properties per product. Unknown or bad events go to a "quarantine" topic instead of polluting data.
- Stream processing (Flink): real-time counters (page views per product per minute) → Redis or an OLAP store.
- Data lake (S3 + Parquet files): all events, partitioned by date and event name.
- Warehouse / OLAP (BigQuery, Snowflake, ClickHouse): DAU, funnels and retention via batch jobs and ad-hoc SQL.
2.2 Architecture Diagram
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
APP["Web / Mobile SDK - batch, retry"] --> COL["Collectors"]
COL --> SR["Schema check"]
SR --> K[("Kafka - events")]
SR -->|"invalid"| QU[("Quarantine topic")]
K --> FL["Stream jobs - real-time counts"]
FL --> RT[("Redis / OLAP - live metrics")]
K --> LAKE[("Data lake - Parquet by date")]
LAKE --> ETL["Batch jobs - DAU, funnels"]
ETL --> WH[("Warehouse")]
DASH["Dashboards"] --> RT
DASH --> WHData ModelData model
Event (common envelope):
event_id (UUID), event_name, product, user_id (nullable), anonymous_id, session_id,
client_ts, server_ts, platform, app_version, geo, properties (JSON)
Data lake layout:
s3://events/product=checkout/event_name=page_view/date=2026-09-19/hour=10/part-*.parquetPartitioning by product, event and date means a query like "page views yesterday" reads only a small slice.
Key FlowsFlows
4.1 Tracking an event
- The app calls
track(). The SDK addsevent_id, timestamps and session info, and queues the event locally. - It flushes a batch to the collector. On failure, it keeps the batch and retries later (bounded by disk space).
- The collector validates the batch against the schema and writes to Kafka, then returns 200.
4.2 Product page view count
Flink reads page_view events, deduplicates by event_id, and counts per product_id per minute. It writes rolling totals to Redis. The product page reads "1,203 people viewed this in the last hour" from Redis (cached).
4.3 Daily metrics
Hourly or daily jobs build clean tables (sessions, daily active users, funnel steps) from the data lake into the warehouse. Dashboards query those tables.
Deep Dive A — Which timestamp do you believe?Deep dive
Every event carries a time from the device that produced it, and devices lie. Phone clocks drift, users set them by hand, and some sit in a drawer for a week before uploading.
Trust the client's timestamp
Take ts from the SDK and use it for everything: session ordering, daily partitioning, the funnel.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
D1["Phone with clock 3 days fast"] --> E1["purchase at 2026-09-23"]
E1 --> P["Daily partition 09-23"]
P --> R1["Tuesday's revenue lands in Friday"]
D2["Phone with clock 2 hours slow"] --> E2["signup after the purchase it preceded"]
E2 --> R2["Funnel shows purchase before signup - step dropped"]A small fraction of skewed clocks is enough to corrupt every daily number, and a partition that has already been computed keeps receiving events for days. Worse, it is silently attackable: a client can claim any time it likes.
Stamp everything at the server
Ignore the client and use the collector's receive time. Now the clock is one trusted clock, partitions close cleanly, and nobody can backdate an event.
This makes the daily numbers right and the sequences wrong. Ten events queued offline on a phone all arrive in the same 50 ms, so server time cannot tell which happened first. A funnel built on it shows purchase → add_to_cart → page_view in whatever order the upload loop happened to flush, and every within-session analysis becomes noise.
Keep both, and use each for what it can do
Record client_ts and server_ts on every event and give them different jobs:
server_tsfor partitioning, retention and anything billed. It is monotonic, trusted and cannot be forged, so daily tables close on time and stay closed.client_tsfor ordering within a session. Even a badly-set clock is usually consistently wrong on one device, so the differences between its own events are reliable even when the absolute value is not.- The gap between them is itself a signal. A device whose skew exceeds a threshold gets flagged, and its events are excluded from timing analyses rather than quietly corrupting them.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
EV["Event - client_ts + server_ts + event_id"] --> SKEW{"Skew above threshold?"}
SKEW -->|"yes"| FLAG["Flag device - excluded from timing analysis"]
SKEW -->|"no"| OK["Accepted"]
OK --> PART["server_ts - daily partition, retention, billing"]
OK --> ORD["client_ts - order within the session"]
OK --> DED["event_id - dedupe: short window in stream, exact in batch"]
DED --> LATE["Batch reprocesses the last 2-3 days for late uploads"]Deduplicate on event_id in both paths — a short window in the stream for the common retry, and an exact pass in batch, which is also what catches the phone that uploads the same batch eleven hours later.
Deep Dive B — Privacy and costDeep dive
- Consent: the SDK doesn't send (or sends only strictly necessary) events until the user consents.
- PII: don't put emails or names in properties. Hash or drop them at the collector.
- Deletion requests: keep a user → partitions index, or rewrite affected partitions in the lake periodically to remove a deleted user's events. Warehouse tables are rebuilt from the cleaned data.
- Cost: keep raw data for 13 months and aggregates longer, sample very high-volume low-value events (e.g., scroll events at 10%), and compress with columnar formats.
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Client sending | Batched with local retry | Saves battery, survives offline | One request per event: simple, wasteful |
| Buffer | Kafka | Decouples collectors from processing, replay | Write straight to the warehouse: fragile, costly |
| Real-time vs batch | Both (stream counters + batch tables) | Fast panels and accurate reports | Batch only: hours of delay |
| Storage | Parquet in a data lake + warehouse | Cheap, fast analytics | Row DB: slow scans, expensive |
Common Follow-up QuestionsFollow-ups
- "How do you compute a funnel?" For each user, order their events by time and check whether they did step 1 → step 2 → step 3 within a window (e.g., 1 day). Warehouses have functions for this, or you can precompute it daily.
- "How do you count unique viewers cheaply?" Use HyperLogLog sketches per page per hour. They merge across hours with about 1% error.
- "Ad blockers drop events?" Use a first-party collector domain, and accept some loss for web analytics.
Wrap-UpWrap-up
A client SDK batches and retries events with unique IDs, collectors validate them against a schema registry and write to Kafka. Stream jobs produce real-time counters, like page views per product, while everything lands in a partitioned data lake that batch jobs turn into DAU, funnel and retention tables. Deduplicate by event ID, reprocess late data, and build in consent, PII handling and deletion from the start.