Problem RestatementProblem
Design the control plane for a fleet of about 100,000 GPU servers (asked at Oracle and NVIDIA). Each host constantly reports its health: heartbeats, GPU temperature, memory errors (ECC), NVLink/network status and disk health. The control plane must:
- keep an up-to-date view of every host,
- detect sick or dead hosts,
- stop sending new jobs to them and move work away (drain),
- trigger repair workflows (reboot, reimage, open a hardware ticket), and
- return fixed hosts to service.
GPUs are expensive, so both idle broken hosts and jobs failing on bad hosts cost a lot of money.
RequirementsRequirements
1.1 Functional
- Ingest heartbeats and health metrics from all hosts.
- A health state per host, with history.
- Automated detection rules (and ML later), plus manual overrides.
- Repair workflows with multiple steps and retries.
- An API for the job scheduler: "which hosts are healthy and free?"
1.2 Non-Functional
- Scale: 100K hosts × a report every 10 seconds = 10K reports/sec, plus detailed metrics.
- Fast detection: under a minute for dead hosts.
- Few false positives: don't drain healthy hosts running expensive training jobs.
- Safety: automation must never drain a large part of the fleet at once by mistake.
1.3 Scale Estimates
- 10K heartbeats/sec, and ~1M metric points/sec (100 metrics per host per 10s).
- Hardware failures at this size are frequent: maybe hundreds of hosts per day need attention.
1.4 API Design
/v1/hosts/{id}/heartbeat{ ts, gpu: [...], ecc_errors, nvlink_ok, running_jobs }/v1/hosts?state=healthy&gpu_type=H100&free=true/v1/hosts/{id}/drain, POST /v1/hosts/{id}/cordon, GET /v1/hosts/{id}/historyHigh-Level ArchitectureArchitecture
2.1 Overview
- Host agent: collects GPU and system health locally (e.g., via NVIDIA DCGM), runs quick self-tests, and sends heartbeats.
- Ingestion: heartbeats to a state service. Detailed metrics go to a time-series DB.
- Host state store: the current state per host in a strongly consistent store (e.g., etcd or a DB with conditional updates), partitioned by host ID.
- Health evaluator: rules like "no heartbeat for 60s", "ECC uncorrectable errors > 0", "GPU fell off the bus", "NVLink down".
- Remediation engine: a workflow engine (e.g., Temporal) that runs repair steps.
- Job scheduler integration: reads healthy capacity and receives drain requests.
2.2 Architecture Diagram
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
H["Host agents - 100K GPUs"] -->|"heartbeats"| ING["Ingestion"]
H -->|"metrics"| TS[("Time-series DB")]
ING --> ST[("Host state store")]
EV["Health evaluator - rules"] --> ST
EV --> TS
EV -->|"unhealthy"| RE["Remediation workflows"]
RE -->|"drain"| SCH["Job scheduler"]
RE -->|"reboot / reimage / ticket"| H
SCH -->|"only healthy hosts"| ST
OPS["Operators UI"] --> STHost State Machine
healthy → suspect → draining → repairing → validating → healthy
↘ (repair failed 3x) → broken (hardware ticket / RMA)- suspect: one signal looked bad. Stop placing new jobs, but don't kill running ones yet. Wait for confirmation (e.g., a repeated check or a second signal).
- draining: ask the scheduler to checkpoint and move running jobs, or wait for them to finish, up to a deadline.
- repairing: run the workflow (reset GPU → reboot → reimage → hardware ticket).
- validating: run burn-in tests (GPU stress, NCCL bandwidth test) before returning to service.
Every transition is a conditional update (only if current state = X and version = V), so two evaluators or an operator can't make conflicting changes.
Key FlowsFlows
4.1 A host stops heartbeating
- The evaluator notices no heartbeat for 60 seconds.
- Before declaring it dead, check whether its whole rack or switch went silent. If many hosts vanish together, it's probably a network problem, so raise a single incident instead of draining hundreds of hosts.
- For a single host: mark it
suspect, thendraining. The scheduler reschedules its jobs from their last checkpoint. - Remediation tries a remote power cycle (through the out-of-band management controller, BMC). If the host comes back and passes validation, it returns to
healthy.
4.2 GPU memory errors
Uncorrectable ECC errors mean results may be wrong. Drain immediately (don't let a training run continue on bad memory), reset the GPU, and if errors repeat, open a hardware replacement ticket.
Deep Dive A — Not draining healthy hostsDeep dive
With 100,000 hosts, a rule that is wrong one time in a thousand drains a hundred good machines a day. Automation at this scale is judged by its false positive rate, not its detection rate.
Drain on any bad signal
A host reports a high GPU temperature or an ECC error, so the control plane drains it and opens a hardware ticket.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
SIG["One transient signal"] --> DR["Drain the host"]
DR --> JOB["Running training job loses a rank"]
JOB --> RESTART["Whole job restarts from the last checkpoint"]
SIG --> NOISE["Sensor blip, driver hiccup, single correctable ECC"]
NOISE --> FLAP["Host returns healthy, gets drained again next blip"]Most single readings are noise — a correctable ECC error is expected and harmless, a temperature spike may be one poll during a burst. Draining is disruptive: on a synchronous training job, removing one rank restarts everything. The automation costs more than the faults it is chasing.
Require the signal to persist
Only act when the condition holds across several consecutive polls, or crosses a threshold over a window.
Transient noise is filtered, which removes most of the false positives. Two categories survive. A flapping host oscillates around the threshold and is drained, returned and drained again. And the control plane still cannot tell a broken host from a broken job — a bad kernel that hangs the GPU looks exactly like failing hardware, so a faulty training run walks through the fleet draining every host it touches.
Two independent signals, hysteresis, and cross-checking against jobs
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
S1["Signal A - ECC threshold"] --> AND{"Two independent signals agree?"}
S2["Signal B - NVLink errors or failed health probe"] --> AND
AND -->|"no"| WATCH["Watch only - no action"]
AND -->|"yes"| JOBCHK{"Same job failing on many hosts?"}
JOBCHK -->|"yes"| BLAME["It is the job - do not drain hosts"]
JOBCHK -->|"no"| DRAIN["Drain, ticket, repair"]
DRAIN --> BURN["Burn-in test"]
BURN --> HEALTHY["Healthy for a sustained period before returning"]- Two independent signals. Demanding agreement between unrelated measurements — say an ECC threshold and a failed health probe — multiplies the false positive rates together instead of adding them.
- Hysteresis on the way back. A host must pass burn-in and stay healthy for a sustained period before returning to service. Different thresholds for leaving and rejoining are what stop flapping.
- Ask whether it is the job. If the same job is failing on many hosts, the hosts are fine. This single check prevents the most expensive failure mode: one bad workload draining a cluster.
- Measure each rule's precision. Track how often a drained host turned out healthy at repair, per rule. Rules whose precision drops get tuned or disabled — and without this number, nobody can tell which rules are earning their disruption.
Deep Dive B — Safety rails for automationDeep dive
- Rate limits: never drain more than, e.g., 2% of the fleet (or 10% of any one cluster) per hour automatically. Beyond that, page a human.
- Blast radius checks: detect correlated failures (same rack, same firmware, same driver version) and pause automation for that group.
- Idempotent workflows: every step can be retried safely, and workflows persist their progress so a control plane restart resumes them.
- Capacity awareness: repair teams and spare parts are limited, so queue hardware tickets by priority (e.g., hosts in the biggest training clusters first).
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| State store | Strongly consistent, conditional updates | No conflicting transitions | Eventual store: races between evaluators |
| Detection | Rules + confirmation + correlation checks | Few false drains | Single signal triggers: flapping, lost jobs |
| Repair | Durable workflow engine | Survives restarts, retries steps | Cron scripts: fragile, no history |
| Scheduler link | Scheduler reads healthy set, drain via API | Clear ownership | Control plane kills jobs directly: loses work |
Common Follow-up QuestionsFollow-ups
- "How do you handle 100K heartbeats?" Partition hosts across evaluator instances by host ID, and keep last-seen times in memory with a periodic flush.
- "Predict failures?" Train a model on metric history (rising correctable ECC errors, temperature trends) to proactively drain hosts before they fail.
- "Firmware or driver rollouts?" Treat them like deployments: canary a few hosts, validate, then roll out in waves, with automatic pause if failure rates rise.
Wrap-UpWrap-up
Agents send heartbeats and GPU health, a consistent state store tracks each host through a clear state machine, and an evaluator with confirmation and correlation checks decides when a host is really sick. A durable workflow engine drains, repairs, validates and returns hosts, while the scheduler only places work on healthy capacity. Rate limits and blast-radius checks keep the automation from ever taking out a large part of the fleet.