An agent incident almost never announces itself as an error. The request returns 200. The tool call succeeds. The run completes in reasonable time. What went wrong is that the agent emailed the wrong customer, refunded the wrong order, or summarized a document that contained instructions and followed them. Your dashboard shows a healthy service because the service is healthy. The decisions were wrong.
That asymmetry is the whole design problem. Conventional services fail loudly at the boundary of a function call. Agents fail quietly inside a sequence of individually valid actions, each taken with whatever permission the runtime gave them. The seven failure modes below account for most production incidents, and each one has a guardrail that catches it before a human has to notice.
Why agent failures evade your existing monitoring
Classical observability assumes a failure is an exception, a non-200, or a latency spike. Agent failures are semantic. The model selected the wrong tool from a valid set, or selected the right tool with a plausible but incorrect argument, or executed a correct plan against a system whose state had already changed. None of those conditions trip an alert.
OWASP's work on LLM applications and on agentic threats maps the same territory from the security side: prompt injection and excessive agency are recurring themes because an agent that can read untrusted text and call tools with real permissions has both the input and the means. NIST's AI Risk Management Framework supplies the governance scaffolding, but it will not tell you where to put the check. That is an engineering decision, and it belongs at the tool boundary.
The seven failure modes
1. Tool misuse
The agent calls a legitimate tool with a destructive argument: a delete where an update was intended, a production database where staging was specified, an amount with an extra zero. Schema validation catches type errors and misses semantic ones. The guardrail is a tool broker that enforces three things per call: the caller's identity, an argument whitelist with business rules, and a blast-radius limit such as a maximum row count or a maximum transaction value. Irreversible operations get a dry-run step and a human confirmation. Write the policy once in the broker, not in the prompt, because prompts are suggestions and brokers are code.
2. Prompt injection
Indirect injection arrives through content the agent reads, such as a support ticket, a web page, a PDF, or the output of another tool. The instruction text is indistinguishable from data unless the architecture separates them. Practical mitigations: label retrieved content with its provenance and keep it out of the instruction region of the prompt; assume any content that reached the model can influence it; remove secrets from the context entirely so there is nothing to exfiltrate; and restrict egress from tools that read untrusted content, so a compromised run cannot post data to an arbitrary URL. Where the risk is high, run untrusted content through a separate model that only reports findings back to the orchestrating agent rather than acting on it.
3. Runaway loops
The agent retries a failing tool, re-plans, retries, and burns through budget or rate limits. Loops also amplify side effects when the tool is not idempotent. Three limits catch nearly all of them: a step budget per task, a wall-clock deadline, and a token budget. Add loop detection that hashes the last few tool calls and aborts on repetition, idempotency keys on any write, and a circuit breaker per downstream dependency. Persist checkpoints so a killed run can resume rather than repeat.
4. Stale context
Long contexts degrade. Chroma's research on context rot measured accuracy as input grows and found performance falls even on simple retrieval tasks, which matches what practitioners see: the agent cites an early fact that a later document contradicted. Separately, the world moves while the context sits still, so a cached account balance is wrong within minutes. Mitigations: keep a structured scratchpad of confirmed state rather than raw transcript, re-fetch authoritative values immediately before any write, attach timestamps to retrieved facts and refuse to act on facts past their TTL, and compact the transcript when it crosses a threshold instead of letting it grow.
5. Permission creep
The agent starts read-only, gains write access for a demo, then keeps a token with administrator scope because revoking it would break something. Every incident then has a larger blast radius than it should. Give each agent a distinct identity, scope tokens per task, and prefer downscoped delegation over long-lived secrets. Run a monthly diff of each agent's effective permissions against its declared purpose, and treat any new permission as a change that needs review. This is ordinary least-privilege hygiene, applied to a principal that nobody thinks of as an employee.
6. Cost blowups
Cost fails in two shapes: a slow leak from verbose prompts and retries, and a spike from a loop or an unexpectedly large input. Set a budget per task and per tenant, enforce it in the runtime rather than in a dashboard, and route easy cases to a cheaper model. Then measure cost per successful task, not cost per call, because that ratio is what your unit economics actually depend on. A kill switch that a support engineer can pull without a deploy is worth more than a forecast.
7. Silent regressions
A model version changes underneath you, a prompt edit fixes one case and breaks four, or a tool's response format shifts slightly and the agent quietly starts making worse decisions. Nothing errors. The only defense is a regression suite built from real tasks with known-good outcomes, run on every prompt, model, or tool change, plus a small canary with a task success metric. Watch the distribution of tool calls too: a sudden shift in which tools the agent picks is an early signal that behavior has moved.
Where guardrails have to live
Four layers, each with a different job. The input layer classifies and labels untrusted content before it reaches the model. The tool layer is the policy enforcement point and does the heaviest lifting, because every consequential action passes through it. The runtime layer owns budgets, deadlines, loop detection, and checkpoints. The output layer validates structure, checks the result against the intended action, and routes high-risk cases to a human queue.
One architectural rule makes all four easier: keep the agent's plan separate from its permissions. The model proposes actions; the runtime decides whether the identity it is holding may perform them. This is the same separation that made service meshes useful, applied to a component that generates its own requests.
The seven modes at a glance
| Failure mode | Earliest visible signal | Primary guardrail |
|---|---|---|
| Tool misuse | Successful calls with unexpected arguments | Tool broker with argument policy and blast-radius caps |
| Prompt injection | Tool calls that follow text in retrieved content | Provenance labeling, no secrets in context, egress allowlist |
| Runaway loops | Repeated identical tool calls; rising step counts | Step, time, and token budgets plus loop detection |
| Stale context | Answers citing superseded facts | TTL on facts, re-fetch before write, transcript compaction |
| Permission creep | Agent tokens with broader scope than declared purpose | Per-task scoped identities and a monthly permission diff |
| Cost blowups | Rising cost per successful task | Runtime budgets, model routing, kill switch |
| Silent regression | Shift in tool-call distribution; falling task success | Trace-derived eval suite gated in CI, canary releases |
Evaluation that survives after launch
Offline benchmarks will not predict production behavior for long, because your tools and data are what shape the agent's decisions. Sample production traces weekly, label outcomes, and promote the interesting failures into a regression suite. Track four numbers: task success rate, cost per successful task, tool error rate, and human escalation rate. Define the success threshold before launch so that a model upgrade has something to fail against, and version prompts like code, with the eval run attached to every change.
What to do next
- Route every tool call through one broker you control, and make irreversible actions require an explicit confirmation step.
- Give each agent its own identity with per-task scopes, then run a permission diff against declared purpose this month.
- Set step, time, and token budgets per task, add loop detection, and confirm the kill switch works without a deploy.
- Pull twenty production traces, label them, and turn the failures into a regression suite that gates prompt and model changes.
- Put cost per successful task and escalation rate on the same dashboard as latency and error rate.