How I built an AI Agent Failure Forensics tool in one sprint

63% of complex AI agent tasks fail silently. No exception thrown. No alert fired. Just a slightly wrong answer, a frozen loop, or a tool call that silently hallucinated its own parameters. Your Datadog dashboard watches a green screen while your pipeline slowly rots.

The problem nobody wants to name

I ran three AI agents in production for two months before I caught on. The failure modes weren't exceptions — they were drift. The agent would complete a task, return plausible output, and the downstream system would happily use it. Nobody noticed until a customer complained three days later.

Classic observability tools were useless. They detect crashes, not quiet wrongness. The failure was architectural: agents that complete without failing, and produce outputs just broken enough to cause damage.

So I built the thing I needed. One sprint. 72 hours. A micro-SaaS called AI Agent Failure Forensics Sprint.

What the sprint delivers

For $750 flat, I take sanitized agent logs, sanitized API traces, and a description of the expected behavior. In return, the buyer gets:

• An incident forensics report — failure modes ranked by severity
• A replay fixture — a deterministic test case reproducing the failure
• A pre-flight contract check — schema validation for every tool-call parameter
• An error-budget metric — an SLO definition for agent reliability
• A failure taxonomy classification — structural vs tactical root causes

Three artifacts. Not a PDF of suggestions. Not a Notion page of "next steps." Real, runnable, traceable outputs.

The concrete example

Here's a real failure I caught in a dev customer's pipeline. Names sanitized, obviously.

The symptom

An AI agent handling customer support triage was occasionally routing tickets to the wrong queue. Not every time. Not predictably. Just enough to create a low-grade support nightmare.

What the forensics found

Between steps 3 and 4 of the agent's 7-step orchestration, there was a silent tool-call with no return-value guard. The agent called a sentiment-analysis tool. The tool returned an empty string when confidence was below threshold — standard behavior, but the agent had no schema validation on the output. It treated "" as a valid sentiment score and routed everything with empty sentiment as "positive."

The dashboard never showed this. The agent completed every step. No errors.

What the sprint delivered

// The replay fixture — deterministic, CI-runnable
const fixture = {
  input: { ticket_body: "Low confidence ambiguous text", confidence_threshold: 0.7 },
  expected: { sentiment: null, routing: "manual_review" },
  actual:   { sentiment: "",         routing: "positive_queue" },  // silent wrong
  guard_added: "if (sentiment === '' || sentiment === null) enforce manual_review"
};

// Pre-flight contract check — now runs before every agent step
tool_schema["sentiment_analysis"].output.required.push("sentiment_nonempty_flag");

One line of schema validation. One missing null check. That's what $2,400 in monthly token waste looked like.

The client now runs the replay fixture in their CI pipeline on every deploy. The forensics report took 48 hours. The fix took one afternoon.

What I'd do differently

I spent the first 12 hours building the wrong thing — a web UI for log ingestion. Nobody wants to paste 80KB of API traces into a browser form. The sprint is async: buyers send a sanitized zip, I do the analysis, I send back artifacts. I deleted the UI and kept the email. Should have started there.

Also: the pre-flight contract check is the most underrated artifact in the deliverables list. It's the only one that prevents future failures. The forensics report is a post-mortem. The contract check is a guard rail. I should have led with that.