IoT / Sensor Telemetry Pipeline

0Problem RestatementProblem

Design a system that collects readings from millions of devices, such as temperature sensors in stores, GPUs in a data center, or smart meters. Each device sends a reading every few seconds. Users want a dashboard that shows the current values in near real time, and charts of history over days or months.

Two things make this harder than it sounds:

• Devices go offline. When they come back, they upload a batch of old readings, so data arrives late and out of order.
• Sometimes the backend must ask a device to do something, such as "send detailed logs for the last hour".

1RequirementsRequirements

1.1 Functional

• Ingest readings: { device_id, metric, value, timestamp }.
• Show the latest value per device (live dashboard).
• Show history charts for any time range.
• Alerts, e.g., "freezer temperature above -10°C for 5 minutes".
• Send commands to devices (collect logs, change sampling rate).

1.2 Non-Functional

• Scale: millions of devices.
• No data loss, even for readings that arrive days late.
• Live view within a few seconds; history queries in about a second.
• Secure: only real devices can send data.

1.3 Scale Estimates

• 5 million sensors × 1 reading every 10 seconds = 500,000 readings/sec.
• Each reading ≈ 50 bytes → 25 MB/sec, about 2 TB/day raw. Time-series compression brings that down about 10x.
• Keep raw data for 30 days, 1-minute averages for 1 year, and hourly averages forever.

1.4 API Design

MQTT is a lightweight messaging protocol made for small devices on unreliable networks.

2High-Level ArchitectureArchitecture

2.1 Overview

• Device gateway (MQTT broker or HTTPS endpoint): authenticates each device with its own certificate and accepts batches.
• Kafka: buffers all readings, partitioned by device_id, so one device's data stays in order.
• Stream processor: validates readings, drops duplicates, updates the "latest value" store, and checks alert rules.
• Latest-value store (Redis): the current reading per device, for the live dashboard.
• Time-series DB (TimescaleDB, InfluxDB, or Cassandra with time buckets): stores history.
• Rollup jobs: compute 1-minute and 1-hour averages.
• Command service: stores commands and delivers them when the device is online.

2.2 Architecture Diagram

%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
    D["Devices / Sensors"] -->|"MQTT / HTTPS batches"| GW["Device Gateway - cert auth"]
    GW --> K[("Kafka - by device_id")]
    K --> SP["Stream processor - validate, dedupe"]
    SP --> R[("Redis - latest value")]
    SP --> TS[("Time-series DB")]
    SP --> AL["Alert engine"]
    TS --> RU["Rollups 1m / 1h"]
    UI["Dashboard"] --> API["Query API"]
    API --> R
    API --> TS
    CMD["Command Service"] -->|"deliver when online"| GW

3Data ModelData model

Time-series table (partitioned by day, clustered by device):
  device_id, metric, ts, value
  primary key ((device_id, day), metric, ts)

Latest value (Redis hash):
  latest:{device_id} → { temp: -18.2, ts: 1726740000 }

Commands:
  command_id, device_id, type, params, status (pending/sent/done), created_at

The (device_id, ts) key makes each reading unique, so writing the same reading twice just overwrites it. Deduplication comes for free.

4Key FlowsFlows

4.1 Normal reading

1. The device sends a batch every 10–60 seconds (batching saves battery and network).
2. The gateway checks the device certificate and puts the batch on Kafka.
3. The processor writes readings to the time-series DB and updates Redis only if the reading is newer than the stored one.

4.2 Device was offline for 6 hours

1. The device stored readings locally and now uploads them in batches, oldest first.
2. The pipeline writes them into the correct past time slots. Nothing special is needed, since each row has its own timestamp.
3. Redis is not overwritten with old values (the "only if newer" check).
4. Rollups for those past hours are marked "dirty" and recomputed.

4.3 Sending a command

The command is saved as pending. When the device connects (or already is), the gateway pushes it. The device replies "done" with a result. If the device never comes back, the command expires.

5Deep Dive A — Readings that arrive out of orderDeep dive

A sensor on a flaky connection buffers for ten minutes and then uploads everything at once. Whether the pipeline uses the reading's own timestamp or the moment it showed up changes every number downstream.

%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
  D["Freezer sensor - offline 10 min"] --> B["Buffers 200 readings at -18 C"]
  B --> U["Uploads them all at 12:10"]
  U --> W["All 200 land in the 12:10 window"]
  W --> A1["12:00-12:09 shows no data - false 'sensor offline' alert"]
  W --> A2["12:10 average computed from 200 stale readings"]

%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
  R["Reading - measured_at"] --> WM{"Before the watermark?"}
  WM -->|"yes"| AGG["Window aggregate - can raise an alert"]
  WM -->|"no"| LATE["Marked late"]
  LATE --> ST[("Storage")]
  LATE --> RE["Rollups recomputed - charts corrected"]
  AGG --> ST

6Deep Dive B — Storage and costDeep dive

• Time buckets: partition data by device and day. A query for "device X, last week" reads 7 small partitions.
• Downsampling: after 30 days, keep only 1-minute averages (with min and max so spikes are not hidden).
• Compression: time-series stores keep only the difference from the previous value, which works very well for slowly changing sensor data.
• Bad data: validate ranges (a freezer cannot be +500°C). Quarantine obviously wrong data, often caused by firmware bugs, instead of storing it.

7Trade-offs & AlternativesTrade-offs

8Common Follow-up QuestionsFollow-ups

• "What if all devices reconnect at once after an outage?" Kafka absorbs the burst. Devices also wait a random delay (jitter) before uploading, so they don't all send at the same second.
• "How do you secure devices?" Each device gets a unique certificate at the factory. If one is stolen, revoke that one certificate only.
• "Real-time and batch analytics?" Send the Kafka stream to both the stream processor (live) and a data lake (batch jobs such as monthly reports).

9Wrap-UpWrap-up

Authenticate each device, buffer batched readings in Kafka by device, and store them in a time-series DB keyed by device and timestamp, which also removes duplicates. Keep the latest value in Redis, handle late uploads with event-time processing and rollup recomputation, downsample old data, and deliver commands through the same gateway when devices are online.