Problem RestatementProblem
Apple asked a data engineer to design the data systems behind a TikTok-like app. Billions of interaction events every day (views, watch time, likes, shares, comments, follows) must be ingested, processed in streaming and batch, and turned into data products: recommendation features, creator analytics, business dashboards and ad reporting. The focus: scalable ingestion, processing orchestration, partitioning, and failure handling.
RequirementsRequirements
- Ingest ~50B events/day (~600K/sec average, much more at peaks).
- Real-time aggregates (views per video in the last minute, trending) within seconds.
- Daily and hourly tables for analytics (DAU, watch time, creator stats) with correct numbers.
- Features for recommendation models (fresh and historical).
- Handle late and duplicate events, backfills, and data quality problems.
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
APP["Apps + backend services"] --> COL["Collectors"]
COL --> K[("Kafka - events by type")]
K --> FL["Flink - real-time aggregates, features"]
FL --> RT[("Real-time stores - Redis, OLAP")]
FL --> FS[("Online feature store")]
K --> RAW[("Lakehouse raw - Iceberg/Delta tables")]
RAW --> ORCH["Orchestrator - Airflow DAGs"]
ORCH --> SPK["Spark batch - clean, dedupe, model"]
SPK --> CUR[("Curated tables - facts, dimensions")]
CUR --> WH["Query engines - Trino / Spark SQL"]
CUR --> OFF[("Offline features / training sets")]
WH --> DASH["Dashboards, creator analytics"]
DQ["Data quality checks"] --> CURLayers (the "medallion" idea, explained simply)
- Raw (bronze): events exactly as received, partitioned by
date/hour/event_type, stored in columnar files (Parquet) in a table format (Iceberg/Delta) that supports schema changes, time travel and safe rewrites. - Clean (silver): deduplicated (by event_id), validated, enriched (user country, video metadata), with bad rows quarantined.
- Curated (gold): business tables such as
fact_video_views_daily (video_id, date, views, watch_seconds, unique_viewers_hll),dim_video,dim_creator, and daily active users.
Streaming Path
- Flink jobs read Kafka and compute windowed aggregates (views per video per minute, trending scores) and real-time features (a user's last 50 interactions, a video's recent completion rate) into the online feature store for recommendations.
- Checkpointing gives exactly-once state within Flink, and outputs are idempotent (upserts).
Batch Path and Orchestration
- Airflow DAGs run hourly and daily: raw → clean → curated → aggregates → exports. Each task depends on its inputs being complete (sensors on partitions).
- Late data: re-process the last 2–3 days' partitions every run (overwrite the partition), so late events are included.
- Backfills: when logic changes, re-run the DAG for past dates. Idempotent partition overwrites make this safe.
- Partitioning and file sizes: partition by date (and hour for big tables), and compact small files into ~512 MB files, since many tiny files make queries slow.
Deep Dive — Noticing that the data is wrongDeep dive
A pipeline that fails loudly is easy. The dangerous case is one that succeeds while producing wrong numbers, and feeds them to dashboards and models for a week.
Alert when a job fails
Monitor task exit codes and page the owner on failure.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
UP["Upstream changes a field from string to int"] --> JOB["Job runs - exits 0"]
JOB --> NULL["Casts fail silently - 40% nulls"]
NULL --> TABLE["Curated table published"]
TABLE --> DASH["Dashboards show a 40% drop in engagement"]
DASH --> WEEK["Discovered a week later by a confused PM"]Exit status measures whether the code ran, not whether the output is right. Most real data incidents are successful runs over bad input, so this catches the least dangerous category.
Data quality checks on curated tables
Assert row counts against yesterday, null rates, key uniqueness and value ranges — and fail the task when a check trips, so downstream jobs do not consume bad data.
This is the core of the answer and it catches the silent corruption above. Its limit is that it detects the breakage after it has propagated into your tables: by the time the check fires, the upstream change already happened and the pipeline has to be rerun.
Contracts upstream, checks in the middle, SLAs downstream
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
PROD["Producer teams"] --> CONTRACT["Data contract - schema registry, no breaking change without a version bump"]
CONTRACT --> RAW[("Raw")]
RAW --> CHECKS["Quality checks: counts vs yesterday, null rates, key uniqueness, ranges"]
CHECKS -->|"fail"| STOP["Stop downstream tasks, alert the owner"]
CHECKS -->|"pass"| CUR[("Curated")]
CUR --> SLA["SLA: daily creator stats ready by 06:00 UTC"]
SLA --> LAG["Lag monitoring - alert before the deadline is missed"]
RAW --> TTL["TTL 90 days raw; curated retained longer; tiered storage"]- Contracts move the failure upstream. A schema registry that rejects a breaking change without a version bump stops the incident at the producer, where it is a build failure rather than a week of wrong dashboards.
- Checks stop propagation. They must fail the task, not just warn — a warning on a dashboard nobody reads is how bad data reaches production anyway.
- An SLA with lag monitoring turns "is it ready?" into something measurable, and alerts before the deadline rather than after consumers notice.
- Retention and cost are part of reliability. Keeping raw data for 90 days is what makes a rerun possible when a bug is found — the recovery path depends on it.
The framing worth stating: the pipeline's contract is with its consumers, and freshness, completeness and correctness are all part of it. A job that finished is not the same as a table that is right.
Wrap-UpWrap-up
Ingest events through collectors into Kafka. A Flink streaming path produces real-time aggregates and online features, while the raw stream lands in a lakehouse (Iceberg/Delta) that Airflow-orchestrated Spark jobs refine from raw to clean to curated tables. Partition by date, compact files, re-process recent partitions for late data, backfill idempotently, and gate every table with data-quality checks and SLAs for the dashboards, analytics and recommendation models downstream.