Problem RestatementProblem
Amazon asked: design the human-avoidance subsystem for autonomous robots in a warehouse where people and robots share space. Each robot must detect humans around it, predict where they're going, and adjust its path or stop fast enough to never hit anyone, while staying productive (not stopping for no reason). This is a safety-critical real-time system.
RequirementsRequirements
- Detect humans (and other obstacles) 360° around the robot, in all lighting.
- Reaction: from detection to braking within a strict budget (e.g., under 100 ms).
- Always able to stop in time given its speed (stopping distance), and speed limits near people.
- Fail safe: if sensors or software fail → stop.
- Minimize false stops (productivity), and log everything for incident review.
Layered ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
SEN["Lidar, cameras, ultrasonic, bumpers"] --> PER["Perception - detect + track humans"]
PER --> PRED["Motion prediction - where will they be in 1-3 s"]
PRED --> PLAN["Local planner - slow, reroute, yield"]
PLAN --> CTRL["Motor control"]
SEN --> SAFE["Certified safety controller - protective zones"]
SAFE -->|"override: stop"| CTRL
FLEET["Fleet manager - zone rules, traffic"] --> PLAN
PER --> LOG[("Event recorder")]Two independent layers:
- Smart layer (perception, prediction, planning): tries to avoid people smoothly, by slowing early, going around, or waiting.
- Safety layer (simple, certified hardware/firmware): a safety-rated lidar with protective zones. If anything enters the inner zone, it stops the motors directly, regardless of what the smart software says. The zone size grows with speed. This layer is the guarantee, and the smart layer is for efficiency.
Perception and Prediction
- Sensor fusion: lidar gives accurate distance, cameras recognize people (ML detector), and ultrasonic and bumpers cover close range. Fuse detections into tracked objects with position, velocity and a "human" confidence.
- Tracking: follow each person over time (a Kalman filter), which smooths noise and gives velocity.
- Prediction: extrapolate short-term paths (1–3 seconds), with uncertainty growing over time. Treat humans as unpredictable and keep bigger margins than for static objects.
Deep Dive — Deciding how fast to move near a personDeep dive
The robot has detected a person and predicted where they are going. Turning that into a speed is where a warehouse robot either is or is not safe.
Stop when something is close
Set a distance threshold and halt when anything is inside it.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
D["Person within 2 m"] --> STOP["Full stop"]
STOP --> FAR["At 2.1 m, still moving at full speed"]
FAR --> LATE["Closing fast - cannot stop in the remaining distance"]
STOP --> BUSY["In a busy aisle - stops constantly"]
BUSY --> USELESS["Throughput collapses; operators start ignoring the robots"]A fixed distance ignores speed: at full speed the stopping distance may exceed the threshold, so the robot is unsafe at the boundary and uselessly timid inside it. A robot that stops constantly also trains people to treat it as harmless, which is its own hazard.
Slow down as distance decreases
Scale speed with the distance to the nearest person.
Motion is smoother and the behaviour more sensible. It is still not a guarantee: the speed curve is tuned by feel, not derived, so there is no statement of the form "the robot can always stop in time" — and that is precisely the claim a safety case has to make.
Derive the speed limit from stopping distance, and layer the zones
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
PRED["Predicted human positions"] --> DIST["Closest predicted distance"]
DIST --> VMAX["v_max = f(distance, braking capability, reaction time)"]
VMAX --> GUAR["The robot can always stop before the closest predicted position"]
VMAX --> WARN["Warning zone - slow down"]
WARN --> PROT["Protective zone - stop - enforced by the safety layer"]
PLAN["Short-horizon planner"] --> AISLE["In an aisle: yield or wait"]
PLAN --> OPEN["In the open: reroute with clearance"]
FLEET["Fleet manager"] --> ZONE["Zones marked 'human work area' - robots slowed or excluded"]- The speed limit is derived, not tuned.
v_maxfollows from braking capability and reaction time, so the safety property is a consequence of the formula rather than a hope about the curve. - Use predicted positions, not current ones. A person walking toward the robot will be closer by the time it reacts; planning against where they are now builds the reaction delay into the margin.
- Two zones, with the inner one enforced independently. The planner slows in the warning zone; the safety layer stops in the protective zone regardless of what the planner decided. As with the traffic-light conflict monitor, the last line of defence must not share code with the thing it is protecting against.
- Yield rather than squeeze. In a narrow aisle the correct behaviour is to wait or back out; rerouting with clearance is for open areas. Attempting to pass in a confined space is how clearance margins get consumed.
Fleet-level zoning completes it: marking areas as human work zones slows or excludes robots there entirely, which is a far stronger guarantee than relying on per-robot perception in the places where people are densest.
Reliability, Validation and Logging
- Watchdogs: if perception output is late or a sensor fails a health check → reduce speed or stop (fail-safe).
- Determinism: fixed-rate control loops, real-time scheduling, and no heavy background work on the safety path.
- Validation: simulation with many human-behavior scenarios, closed-course tests, and a documented safety case (following industrial safety standards).
- Event recorder: log sensor snapshots around every stop or near-miss for analysis and model improvement.
Wrap-UpWrap-up
Fuse lidar, camera and close-range sensors to detect and track humans, predict their short-term motion, and let a local planner cap speed by stopping distance and slow, yield or reroute early. Put an independent, certified safety layer underneath, with speed-dependent protective zones that cut the motors directly. Add watchdogs that fail safe, fleet-level zone rules, simulation-heavy validation, and an event recorder for every stop and near-miss.