Problem RestatementProblem
Design the camera perception pipeline for a self-driving car (asked at NVIDIA). Several cameras (e.g., 8, around the car) capture images many times per second. The pipeline must turn them into a list of objects (cars, pedestrians, cyclists, lanes, traffic lights), with position, speed and uncertainty, and hand that to the planning system within a strict time limit, running on the car's own computer. Trace the data from physical cameras to the planner.
RequirementsRequirements
- 8 cameras at 30 frames per second, high resolution.
- Detect and track objects around the car, and estimate distance and velocity.
- End-to-end latency (photon to planner) under ~100 ms, with small jitter.
- Keep working safely if a camera fails or a stage is late.
- Log data for offline training and debugging.
1.1 Rough Budget (example)
| Stage | Time |
|---|---|
| Capture + transfer | ~10 ms |
| Preprocess (debayer, resize, undistort) | ~5 ms |
| Neural network inference | ~30 ms |
| Post-processing + fusion + tracking | ~15 ms |
| Handoff to planning | ~5 ms |
| Total | ~65 ms (leaving margin) |
Pipeline
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
CAM["8 cameras - hardware-triggered"] --> CAP["Capture - timestamps"]
CAP --> PRE["Preprocess on GPU - undistort, resize"]
PRE --> NN["Perception models - detection, segmentation, depth"]
NN --> POST["Post-process - NMS, 3D boxes"]
POST --> FUS["Fusion - cameras + radar/lidar"]
FUS --> TRK["Tracking - object IDs, velocity"]
TRK --> PLAN["Planning"]
CAP --> LOG[("Data logger - selected clips")]
HM["Health monitor"] --> PLANKey Stages Explained
- Synchronized capture: all cameras are triggered by hardware at the same instant and stamped with a shared clock (e.g., PTP time sync). Without this, objects appear in different places in different cameras.
- Calibration: each camera's intrinsics (lens) and extrinsics (position and angle on the car) turn pixels into 3D rays. Calibration is checked online, since cameras shift slightly over time.
- Preprocessing on GPU: convert raw sensor data, fix lens distortion, resize and normalize. Keep data on the GPU (zero-copy) to avoid slow memory transfers.
- Inference: a multi-camera model (e.g., bird's-eye-view networks) detects objects, lanes and traffic lights, and estimates depth. Models are optimized (TensorRT, lower precision like FP16/INT8) and batched across cameras.
- Post-processing: remove duplicate boxes (non-max suppression) and produce 3D boxes with confidence scores.
- Fusion: combine with radar (good speed measurement) and lidar (good distance), if present.
- Tracking: a tracker (e.g., Kalman filter) links detections over time into objects with IDs and velocities, and smooths noise.
- Output: an object list with timestamps and uncertainty, sent to planning.
Deep Dive — Hitting a deadline on every frameDeep dive
Eight cameras at 30 frames a second. Perception does not need to be fast on average; it needs to be finished before the deadline, every time, or the planner steers using a stale picture of the world.
Queue frames and process them in order
Frames go into a queue; the pipeline takes the next one when it finishes the last.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
C["8 cameras x 30 fps"] --> Q["Frame queue"]
Q --> P["Perception - occasionally slow"]
P --> LAG["Queue grows during a heavy scene"]
LAG --> OLD["Planner receives a detection from 400 ms ago"]
OLD --> WRONG["Acts on where the pedestrian used to be"]A queue converts an overload into latency, which is the wrong currency here. The system never drops anything and never catches up, so the further behind it falls, the more dangerously wrong its output becomes — while every frame is still processed "successfully".
Bound the queue and drop the oldest
Cap the queue at one or two frames and discard the oldest when it overflows. The pipeline always works on something recent, and the backlog cannot grow without limit.
Freshness is largely restored. What is still unbounded is the variance: a garbage collection pause, a dynamic allocation that hits the allocator slow path, or a logging thread that happens to get scheduled can each blow a single frame's budget. The average is fine and the worst case — the only case that matters — is unknown.
Make the deadline the contract
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
F["Frame with capture timestamp"] --> D{"Can it finish by its deadline?"}
D -->|"no"| DROP["Drop it - take the next one"]
D -->|"yes"| RUN["Run on pinned cores and reserved GPU streams"]
RUN --> PRE["Pre-allocated buffers - no allocation in the loop"]
RUN --> OUT["Objects, each stamped with the capture time"]
OUT --> PLAN["Planner extrapolates from capture time to now"]
LOG["Logging, telemetry"] --> LOW["Lower priority - can never preempt perception"]- Drop rather than delay. A late frame is worthless; the next one is already better. Stating this explicitly is what separates a real-time design from a throughput design.
- Reserve the hardware. Pin perception to specific CPU cores and GPU streams so nothing else — logging, diagnostics, map updates — can take time from the critical path.
- Make execution time bounded. Pre-allocate every buffer, avoid dynamic allocation and garbage collection inside the loop, and cap any variable-length work. Determinism matters more than peak speed.
- Timestamp everything with capture time, not processing time. The planner knows the pipeline took 80 ms and extrapolates each object forward to now — so a known, bounded latency is correctable, while an unknown one is not.
That last point is the reason the whole design is built around a fixed budget: predictable latency can be compensated for; variable latency cannot.
Safety and Degraded Operation
- Health monitor: checks every stage's latency and output sanity (e.g., too few detections, frozen images). If a camera fails, mark its field of view as "unknown" and tell planning, which slows down or pulls over.
- Redundancy: overlapping camera views, plus radar/lidar, so no single sensor failure leaves a blind spot. Critical compute may be duplicated.
- Fail-safe: if perception stops producing valid output within the deadline, planning triggers a minimal-risk maneuver.
Data Logging for Training
- Continuously record into a ring buffer. When something interesting happens (a hard brake, disagreement between sensors, the driver taking over), save the clip plus metadata. You can't upload everything.
- Upload saved clips when the car is parked and on Wi-Fi. They feed labeling, retraining and regression tests.
Wrap-UpWrap-up
Trigger all cameras together on a shared clock, calibrate them, preprocess on the GPU, run optimized multi-camera models, then post-process, fuse with other sensors and track objects over time, all inside a ~100 ms budget. Treat it as a real-time system (drop late frames, fixed priorities, no dynamic allocation, timestamps everywhere). Add a health monitor, sensor redundancy and fail-safe behavior, and log interesting clips from a ring buffer for training.