Problem RestatementProblem
An AI agent calls external tools (search, databases, APIs). Sometimes a call fails, but the error the user sees doesn't show where it failed:
- while preparing the request (the model produced bad arguments),
- while reaching the tool (network, DNS, auth),
- while the tool was executing (the tool's own error or timeout),
- while parsing the response (unexpected format),
- or during retries (the first error hidden by the last one).
Design tracing so engineers can see exactly which stage failed and why. TikTok asked this.
RequirementsRequirements
- Every agent run produces a trace: model calls, tool calls, and their stages as nested spans.
- Each span records the start and end time, status, error type and message, and (safely) inputs and outputs.
- Retries are linked to the same logical tool call, with each attempt visible.
- Traces are searchable ("all failed
search_docscalls in the last hour with PARSE errors"). - Sensitive data is redacted, and payload sizes are limited.
- Low overhead on the agent.
Trace Structure
Trace: agent_run (trace_id=abc)
├─ span: llm_call #1 (tokens, latency)
├─ span: tool_call search_docs (call_id=7)
│ ├─ span: prepare_args ok (validated JSON schema)
│ ├─ span: attempt 1
│ │ ├─ span: send ok (DNS 2ms, connect 10ms)
│ │ ├─ span: execute ERROR timeout after 5000ms (from tool's own span)
│ ├─ span: attempt 2
│ │ ├─ span: send ok
│ │ ├─ span: execute ok (812ms)
│ │ └─ span: parse_response ERROR missing field "results"
│ └─ result: FAILED (stage=parse, attempts=2)
└─ span: llm_call #2 ...Each span has trace_id, span_id, parent_span_id, name, stage, status, error.type, attributes (tool name, attempt number, HTTP status, sizes).
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
AG["Agent runtime - SDK creates spans"] -->|"trace context header"| TOOL["Tool service - continues the trace"]
AG --> COL["Collector - redact, sample, batch"]
TOOL --> COL
COL --> K[("Kafka")]
K --> ST[("Trace store - columnar / search")]
K --> MET["Metrics: failure rate per tool and stage"]
MET --> AL["Alerts"]
UI["Trace viewer UI"] --> ST- Instrumentation SDK (built on OpenTelemetry): wraps each tool call and automatically creates stage spans. Engineers don't have to remember to add them.
- Context propagation: the SDK sends
traceparentheaders to tools. Tools that are instrumented add their own internal spans (e.g., the DB query inside), so "execute" failures show the tool's side too. - Collector: redacts secrets and personal data (API keys, emails), truncates payloads (e.g., keeps the first 4 KB plus a hash of the full body), and batches exports.
- Sampling: keep 100% of traces with errors and a small percentage of successful ones (tail-based sampling: decide after the run finishes).
- Trace store: a columnar or search store for queries by tool, stage, error type and time.
Deep Dive — Making "the tool call failed" actionableDeep dive
An agent calls a tool and something goes wrong. The only question that matters operationally is whose bug is it, and a raw error message almost never answers it.
Log the exception
Catch whatever was thrown and write its message.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
E["Exception logged"] --> M1["'Connection reset by peer'"]
E --> M2["'Invalid argument: date'"]
E --> M3["'Unexpected token in JSON'"]
M1 --> WHO["Network? Tool? Model? Our parser?"]
M2 --> WHO
M3 --> WHO
WHO --> DASH["Dashboard says: 4.2% of tool calls fail"]
DASH --> NOACT["Nobody knows which team should act"]Every failure mode collapses into one number. The error text is written by whichever library happened to throw, so grouping it produces categories that reflect implementation details rather than causes.
Catch per stage and label it
Wrap each step separately — building arguments, making the request, parsing the response — and tag the error with the stage it came from.
This is most of the value, and it is cheap. The stage tells you roughly where to look. What is still missing is the distinction between stages that mean genuinely different things: a tool that returned a business error and a tool that timed out are both "the call stage", and they belong to different owners with different fixes.
An error taxonomy that maps to an owner
Make the type explicit, and choose the categories so each one names who fixes it:
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
CALL["Tool call - a span per stage"] --> P["PREPARE - schema validation failed"]
CALL --> N["NETWORK - DNS, connect, TLS, proxy 5xx"]
CALL --> T["TOOL_ERROR - tool returned an error or timed out"]
CALL --> PA["PARSE - response did not match the schema"]
CALL --> RX["RETRY_EXHAUSTED - summary linking every attempt"]
P --> O1["Owner: prompt / tool description"]
N --> O2["Owner: infrastructure"]
T --> O3["Owner: the tool's team"]
PA --> O4["Owner: contract drift between us and the tool"]Each category is a different action, which is the test of a good taxonomy:
PREPAREmeans the model produced arguments the schema rejected — fix the tool description or the prompt. A rise here usually follows a model or prompt change, not an infrastructure one.NETWORKnever reached the tool at all, so paging the tool's team wastes everyone's time.TOOL_ERRORreached it and it said no. That is the tool owner's, and the tool's own traces continue the story.PARSEmeans the response is valid but not the shape we expected — contract drift, which is the failure that appears without anyone deploying anything on our side.RETRY_EXHAUSTEDis the summary a human reads first, and it must link to every attempt, because the interesting pattern is usually that the attempts failed for different reasons.
Emit a span per stage with structured attributes rather than a message string, so the dashboard can group by cause and by tool without parsing text — and so the argument that failed validation is attached to the span that rejected it.
Using the Data
- Dashboards and alerts: failure rate per tool and stage. Alert if
search_docsPARSE errors jump after a deploy. - Debugging: open a trace to see the exact arguments, timings and responses (redacted) per attempt.
- Replay: re-run a failed tool call with the same inputs in a sandbox.
- Evals: turn common failures into test cases for prompts and tools.
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Format | OpenTelemetry spans per stage | Standard, cross-service | Free-form logs: hard to correlate |
| Sampling | Tail-based, keep all errors | Debuggability at low cost | Keep everything: expensive |
| Payloads | Redacted + truncated + hashed | Safe and useful | Full payloads: privacy risk; none: can't debug |
| Retries | Attempts as child spans of one call | Clear history | Only last error: hides root cause |
Wrap-UpWrap-up
Wrap every tool call in an SDK that emits a span per stage (prepare, send, execute, parse) and per retry attempt under one logical call, propagate the trace context into the tools so their own spans join the trace, and classify errors by stage. A collector redacts and truncates payloads and keeps all error traces with tail-based sampling. The trace store powers a viewer, per-tool and per-stage failure metrics with alerts, replays and eval datasets.