Smart Grid Control over Unreliable Devices

0Problem RestatementProblem

Design a central system that watches and controls millions of smart devices on a power grid: EV chargers, air conditioners, fridges, water heaters and hospital equipment. All of them talk to us over the unreliable public internet. Devices report their power use and status, and those reports can be delayed, duplicated or out of order. When total load gets too high, the system sends commands like "reduce all ACs in region X by 20%". Some devices are offline, some users have opted out or overridden, and critical devices (hospital equipment) must never be turned down. Finally, the system must reconcile: which devices actually did what we asked?

OpenAI asked several versions of this (monitoring, controller, command dispatch, "underspecified power-plant prompt"). The skills tested are handling unreliable networks and designing safe control loops.

1RequirementsRequirements

1.1 Functional

• Ingest device telemetry: power draw, state and settings.
• A live view of load per region.
• Policies: rules that decide when and how to reduce load, respecting device type, user preferences and exemptions.
• Send commands to groups of devices, and track acknowledgements and actual effect.
• Reconcile the expected vs actual response, and retry or escalate.

1.2 Non-Functional

• Safety first: never curtail exempt devices, respect user overrides, and cap how much and for how long a device is curtailed.
• Correct under unreliable delivery: duplicates, reordering, long offline periods.
• Timely: react to overload within seconds to a minute.
• Scale: millions of devices.

1.3 Scale Estimates

• 5M devices reporting every 30 seconds → ~170K reports/sec.
• A regional event: a command to 500K devices within a minute.

1.4 API Design

• Device → cloud: POST /telemetry (or MQTT) { device_id, seq, ts, watts, state, desired_version_applied }
• Cloud → device: MQTT topic devices/{id}/desired → { version, target: { max_watts: 1200 }, valid_until }
• Operator: POST /v1/events { region, reduce_pct: 20, duration_min: 30, device_types: ["ac"] }

2High-Level ArchitectureArchitecture

2.1 Overview

• Device gateway (MQTT): authenticates each device with a certificate and keeps connections open.
• Telemetry pipeline: Kafka → stream processor → per-device latest state + per-region load aggregates.
• Policy engine: watches load and forecasts, and decides targets per device group, filtering out exempt and opted-out devices.
• Device shadow / desired-state store: for each device, the desired state (what we want) and the reported state (what it says it did).
• Command dispatcher: pushes desired-state changes, and retries until acknowledged or expired.
• Reconciler: compares expected vs measured load reduction.

2.2 Architecture Diagram

%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
    D["Devices"] <-->|"MQTT"| GW["Device Gateway"]
    GW --> K[("Kafka - telemetry")]
    K --> SP["Stream processor - dedupe, order by seq"]
    SP --> SH[("Device shadows - reported state")]
    SP --> AG[("Region load aggregates")]
    PE["Policy Engine - safety rules"] --> AG
    PE --> SH
    PE -->|"new desired state"| DS[("Desired state store")]
    DS --> CD["Command Dispatcher - retry until ack"]
    CD --> GW
    RC["Reconciler"] --> SH
    RC --> DS
    RC --> AG

3Handling Unreliable Reports

• Every report carries a device sequence number and the device's timestamp.
• Duplicates: ignore reports with a seq we've already processed for that device.
• Out of order: only update "latest state" if seq is newer. Older reports still go to history storage at their own timestamp.
• Late data for aggregates: region load uses event time with a short watermark (e.g., 30 seconds), and marks regions with many silent devices as "uncertain" instead of treating silence as zero load.
• Clock problems: device clocks drift, so the server records the receive time too, and flags big differences.

4Sending Commands Safely: Desired State, not "Do X Now"

Instead of fire-and-forget commands, use a desired-state model (like AWS IoT device shadows or Kubernetes):

• We set desired = { version: 18, max_watts: 1200, valid_until: 15:30 } for each device (or group).
• The device applies it and reports applied_version: 18.
• Idempotent: resending version 18 changes nothing, and a device ignores versions older than what it already has.
• Offline devices: when they reconnect, they fetch the latest desired state. If valid_until has passed, it no longer applies, so an old curtailment never kicks in hours late.
• Expiry built in: every curtailment has an end time, so a device that loses connection returns to normal by itself (a safety default).

5Deep Dive A — Turning down a million devices without hurting anyoneDeep dive

A curtailment event reaches EV chargers, thermostats and, if nobody stops it, hospital equipment. The safety question is not "can we send the command" but "what happens when the server is wrong".

%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
  PE["Policy engine - exemptions, priority groups"] --> CMD["Curtail command"]
  CMD --> DEV["Device firmware"]
  DEV --> C1{"Am I exempt? - medical class"}
  C1 -->|"yes"| REJ["Refuse and report"]
  C1 -->|"no"| C2{"Inside comfort bounds and daily limits?"}
  C2 -->|"no"| CLAMP["Clamp to the safe value"]
  C2 -->|"yes"| APPLY["Apply, report state"]
  DEV --> OVR{"User pressed override?"}
  OVR -->|"yes"| REJ

6Deep Dive B — Reconciliation and verificationDeep dive

• For each event, compute the expected reduction: sum of targeted devices × their expected drop.
• Measure the actual reduction from telemetry and region meters.
• Devices that didn't acknowledge, or acknowledged but didn't reduce, get a retry, and if they keep failing they're marked unreliable (and less relied on next time).
• If the region is still over target, the policy engine recruits more devices from the next priority group.
• Everything is logged for audits and for settlement (customers may be paid for participating).

7Trade-offs & AlternativesTrade-offs

8Wrap-UpWrap-up

Ingest telemetry through an authenticated MQTT gateway into Kafka, deduplicate and order it with per-device sequence numbers, and keep device shadows and region load aggregates. Let a policy engine with strict safety rules choose targets, and express commands as versioned, expiring desired states that devices apply idempotently, even after being offline. Roll changes out in waves, and continuously reconcile expected vs actual load reduction, retrying or recruiting more devices when needed.