Problem RestatementProblem
Anthropic asked: design the safety and reliability layer around an LLM-powered production feature (e.g., a support assistant). Models can receive malicious inputs (prompt injection), produce harmful or wrong outputs, return invalid formats, be slow, or fail entirely. Design guardrails (checks before and after the model) and fallbacks (what to do when something fails), without making the product slow or annoying.
The Request Pipeline
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
U["User input"] --> IN["Input guardrails - PII, injection, policy"]
IN -->|"blocked"| REF["Safe refusal / help message"]
IN --> CTX["Build prompt - system rules + retrieved data"]
CTX --> M1["Primary model - timeout"]
M1 -->|"error / timeout"| FB["Fallback: retry, smaller model, cached answer"]
M1 --> OUT["Output guardrails - schema, safety, grounding"]
FB --> OUT
OUT -->|"fails"| REPAIR["Repair / regenerate / safe template"]
OUT --> RESP["Response to user"]
OUT --> LOG[("Logs + metrics")]Input Guardrails
- Validation: length limits, allowed languages, rate limits per user.
- PII handling: detect and mask personal data (card numbers, IDs) before sending to the model or logs, when not needed.
- Prompt injection detection: classifiers and heuristics for "ignore previous instructions", and hidden instructions in retrieved documents. Treat retrieved content as data (clearly delimited), never as instructions.
- Policy checks: disallowed topics for this product, with a friendly refusal.
Output Guardrails
- Format validation: if the app expects JSON, validate it against a schema. If invalid, try a repair (ask the model to fix it, or use constrained decoding), then fall back.
- Safety filters: toxicity, self-harm, and leaked secrets or PII in the output.
- Grounding checks (for RAG): does the answer cite retrieved sources? Are claimed facts supported? If not, answer "I'm not sure" or show sources only.
- Business rules: never promise refunds, never quote prices not in the data, and so on.
Deep Dive — What the feature does when the model does not answerDeep dive
The model times out, rate-limits, or returns something the output checks reject. A support assistant cannot simply fail — that is the moment it exists for.
Retry the same model
On error, call again.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
REQ["Request"] --> M["Primary model"]
M -->|"429 or 5xx"| R1["Retry"]
R1 --> M
M -->|"still failing"| R2["Retry again"]
R2 --> M
M --> LOAD["Retries multiply load on a model already overloaded"]
LOAD --> USER["User waits through every attempt, then sees an error"]Retrying into a rate limit makes the rate limit worse, and the user pays for each attempt in latency before getting nothing. During a provider incident this turns a degraded feature into an unusable one.
A strict timeout and one bounded retry
Cap each call, retry once with backoff for transient errors, then give up.
The right first step: the latency is now bounded and the retry storm is gone. But "give up" is still an error message on the support page. The feature has no degraded mode — it is either fully working or absent, and most of the value is in the middle.
A fallback chain, guarded by a circuit breaker
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
REQ["Request"] --> CB{"Circuit breaker - primary healthy?"}
CB -->|"open"| SMALL["Skip straight to the fallback"]
CB -->|"closed"| P["Primary model - strict timeout, one retry"]
P -->|"fails"| SMALL["Smaller or faster model"]
SMALL -->|"fails"| CACHE["Cached answer for a common question"]
CACHE -->|"miss"| TMPL["Templated response + relevant help article"]
TMPL --> HUMAN["Offer handoff to a person"]- Each step is a real answer, not an error. A smaller model's reply, a cached answer to a frequent question, a help article — all are more useful than "something went wrong", and the last rung is an honest handoff rather than a dead end.
- The circuit breaker stops the waiting. When the primary's error rate spikes, requests skip it entirely for a cool-down instead of each one paying the timeout before falling through. That is what keeps latency sane during an incident.
- Streaming needs its own rule. Output checks on a streamed answer run over chunks, or over a buffer of a few sentences before display. Showing text and then retracting it is worse than a short delay — the user has already read it.
State the principle: degrade along a chain, never off a cliff. Each fallback is cheaper and less capable than the one before, and the user is told what they are getting when it matters.
Measuring and Improving
- Metrics: guardrail hit rates per check, false positive rate (from user appeals and reviews), fallback rate, latency added per stage, and incidents.
- Evaluation sets: red-team prompts (injection, jailbreaks), normal prompts (to catch over-blocking), and format tests. Run them on every change.
- Safe rollout: new guardrail versions run in shadow mode first (log what they would block), then gradually enforce.
- Ownership: policies are config (versioned), owned by trust and safety plus product, and applied consistently across features.
Trade-offsTrade-offs
- Stricter checks → safer but more false refusals and added latency. Tune per product and risk level.
- Checking with another LLM is accurate but costly and slow. Use cheap classifiers first, and escalate only uncertain cases.
Wrap-UpWrap-up
Wrap the model in a pipeline: input guardrails (validation, PII masking, injection detection, policy), a carefully built prompt that treats retrieved text as data, and output guardrails (schema validation with repair, safety filters, grounding and business rules). Handle failures with timeouts, one retry, a fallback chain (smaller model, cache, template, human) and circuit breakers, and keep it trustworthy with red-team evals, shadow-mode rollouts and metrics on blocks, false positives and fallbacks.