Problem RestatementProblem
Design an internal monitoring platform like Datadog or Prometheus. Every server and service sends numbers over time, called metrics: CPU usage, request count, error rate, latency. Engineers look at these on dashboards and get alerts when something goes wrong (for example, "error rate above 5% for 5 minutes").
Each metric has a name and tags (labels), such as http_requests{service=checkout, region=us-east, status=500}. The system must take in a huge number of data points, store them cheaply, and answer queries fast.
RequirementsRequirements
1.1 Functional
- Collect metrics from agents on every host and from application libraries.
- Store time series and query them by name, tags and time range, with functions like sum, average and 99th percentile.
- Dashboards with charts that refresh automatically.
- Alert rules that notify on-call engineers through PagerDuty, Slack or email.
1.2 Non-Functional
- Ingest a lot: millions of data points per second.
- Fast queries: a dashboard over the last hour should load in under a second.
- Very reliable: when production is on fire, monitoring must still work.
- Cheap long-term storage: keep data for months, just with less detail.
1.3 Scale Estimates
- 100,000 hosts × 500 metrics each, sent every 10 seconds → 5 million data points/sec.
- Each point is a timestamp + value (16 bytes raw). Time-series compression brings it down to about 1.5 bytes, which is about 650 GB/day.
- Retention: raw 10-second data for 15 days, 1-minute rollups for 3 months, 1-hour rollups for 2 years.
1.4 API Design
/v1/metrics(from agents, batched): [{ name, tags, timestamp, value }, ...]/v1/query?q=avg(cpu{service=checkout}) by (region)&from=-1h&step=60s/v1/alertswith { query, condition: "> 0.05", for: "5m", notify: ["pagerduty:checkout"] }High-Level ArchitectureArchitecture
2.1 Overview
- Agent on each host: collects metrics, adds up counters locally for 10 seconds, and sends batches. It buffers on disk if the backend is unreachable.
- Ingestion gateway: authenticates and validates data, then writes it to Kafka.
- Kafka: a buffer that protects storage from spikes and lets several consumers read the same data.
- Time-series DB (TSDB): stores the data, sharded by series. Examples include Prometheus/Thanos, M3, VictoriaMetrics and InfluxDB.
- Rollup jobs: build the 1-minute and 1-hour summaries.
- Query service: reads from the TSDB shards and merges the results.
- Alert evaluator: runs every alert query on a schedule and sends notifications.
2.2 Architecture Diagram
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
A["Agents on hosts"] --> G["Ingestion Gateway"]
G --> K[("Kafka")]
K --> W["TSDB Writers"]
W --> T[("Time-series DB - sharded")]
K --> RU["Rollup jobs"]
RU --> CS[("Long-term store - object storage")]
D["Dashboards"] --> Q["Query Service"]
Q --> T
Q --> CS
AE["Alert Evaluator"] --> Q
AE --> N["Notifier - PagerDuty, Slack"]Data ModelData model
A series = metric name + a unique set of tags. Each series gets an ID, and its points are stored together in time order.
series index: series_id 981 → cpu_usage{host=web-12, service=checkout, region=us-east}
inverted index: service=checkout → [981, 982, 1044, ...] (to find series by tag)
data blocks: series 981, 12:00–14:00 → compressed [(t, v), (t, v), ...]Key FlowsFlows
4.1 Write path
- The agent sends a batch every 10 seconds.
- The gateway puts it on Kafka, partitioned by
hash(series). - TSDB writers append points to an in-memory block for 2 hours and write a write-ahead log (a file we append to first, so nothing is lost if the writer crashes). Then they flush a compressed block to disk or object storage.
4.2 Query path
- The query service uses the inverted index to find matching series (
service=checkout). - It fetches blocks from the right shards: recent data from memory, older data from disk or rollups.
- It computes the aggregation (e.g., average by region) and returns points for the chart.
4.3 Alert path
Every 30–60 seconds, the evaluator runs each rule's query. If the condition is true for the whole for period, it fires once and sends a single notification (not one per evaluation). When the condition clears, it sends "resolved".
Deep Dive A — High cardinality, the way monitoring diesDeep dive
Cardinality is the number of unique time series. http_requests{service, region, status} with 40 services, 5 regions and 8 status codes is 1,600 series — nothing. Add one more tag and the arithmetic turns on you.
Store whatever tags arrive
The agent sends a metric with its labels, the backend indexes every combination. No limits, no pushback, and it works beautifully until someone adds user_id.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
T1["service x region x status - 1,600 series"] --> OK["fine"]
T2["...plus user_id - 100M users"] --> BOOM["160 billion series"]
BOOM --> IDX["Index no longer fits in memory"]
IDX --> DOWN["Ingestion stalls - every dashboard goes blank"]One team's one-line change takes down monitoring for the whole company — during the incident that made them add the tag. Worse, the damage is done at write time, so there is nothing to roll back.
Put a hard limit on series per metric
Cap each metric at, say, 100K series and reject writes past it. The index stays bounded and the platform survives.
It survives by breaking somebody, silently and arbitrarily: whichever series happen to arrive after the cap are dropped, so a team's dashboard goes half-empty with no explanation and no way to tell which data is missing. The limit protects the platform, not the users of it.
Budgets per team, visible, with somewhere else to send the data
Three parts, and the third is the one that actually solves it:
- A budget per team and per metric, enforced at ingestion, so one team's mistake cannot spend another team's capacity.
- Show them the meter. A page listing each team's top metrics by series count, and an alert when they cross 80% of budget. Teams fix their own cardinality when they can see it; they cannot when it fails silently.
- Route per-entity data where it belongs.
user_id,request_idandtrace_idare not metrics. Metrics answer "how many, how fast, how often" across a population; logs and traces answer "what happened to this one request". Saying that clearly in the docs prevents more incidents than any limit.
When a cap is hit, reject the new series and keep the existing ones, so a dashboard that worked yesterday still works today. Tell the team which metric was rejected and why — an error with a name is a fix, an error without one is a ticket.
Deep Dive B — Keeping monitoring up at 99.99%Deep dive
- Separate failure domains: monitoring must not share databases or clusters with the systems it watches. Otherwise the outage it should report also takes monitoring down.
- Replicate writes to 2 TSDB replicas. Queries can use either one.
- Agents buffer locally so a short backend outage creates a gap that fills in later instead of losing data.
- Meta-monitoring: a tiny, separate system watches the monitoring system ("no data from region X for 2 minutes" is itself an alert).
- Old clients with different metric names: map known naming variants to one canonical name at ingestion, using a rename table, so dashboards stay correct.
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Collection | Push from agents | Works for short-lived jobs, easy to buffer | Pull/scrape (Prometheus): simpler targets, harder at huge scale |
| Buffer | Kafka before TSDB | Absorbs spikes, supports replay | Direct writes: fewer parts, fragile under load |
| Storage | Purpose-built TSDB + rollups | 10x compression, fast range scans | General DB (Cassandra, Postgres): flexible, much costlier |
| Old data | Downsample to 1m/1h | Cheap long retention | Keep raw forever: expensive |
Common Follow-up QuestionsFollow-ups
- "How do you compute p99 latency across 100 hosts?" You cannot average percentiles. Send histograms (counts per latency bucket), add the buckets together, then compute p99 from the merged histogram.
- "How do you avoid alert spam?" Group alerts (one page per service, not per host), add a
forduration, and silence alerts during planned maintenance. - "Multi-tenant?" Add a tenant ID to every series, enforce per-tenant limits, and keep query costs isolated.
Wrap-UpWrap-up
Agents batch metrics and send them through a gateway into Kafka. Writers store compressed time series in a sharded TSDB, and rollup jobs keep cheap long-term summaries. A query service answers dashboards, and an alert evaluator runs rules on a schedule. Control cardinality, use histograms for percentiles, and keep monitoring in its own failure domain so it works when everything else is broken.