AI LAB / Agents / Evaluation
Agent Tools Under Failure
Testing agent recovery when tools time out, reject input, duplicate effects, or return partial data.
AgentsEvaluationReliability
- EXPERIMENT
- EXP-015
- STATUS
- complete
- DATE
- Aug 21, 2026
- TAGS
- Agents · Evaluation · Reliability
EXPERIMENT / HYPOTHESIS
Explicit typed failure states will outperform natural-language error messages in multi-step agent recovery.
- 01Create 120 deterministic failure scenarios
- 02Run identical tasks with typed and prose-only errors
- 03Score completion, unsafe retries, and wasted calls
- Pass rate
- 86.4%
- Unsafe retries
- 2.1%
- Tests
- 120
CONCLUSIONTyped failures improved successful recovery and sharply reduced repeated side effects. The remaining failures were mostly planning mistakes rather than tool handling.
Failure matrix
Scenarios covered validation, authorization, timeout-before-effect, timeout-after-effect, partial result, and dependency outage.
- Retryable
- Terminal
- Unknown outcome
Finding
The agent recovered best when the orchestrator supplied a small stable error code, retry guidance, and confirmed operation state.
- Less prompt ambiguity
- Safer retries
- Easier evaluation