The agent worked 19 times. run 20 booked the wrong thing.
Composite scenario, but based on the kind of failure that keeps showing up in appointment agents.
User said:
book it for next Friday afternoon
The agent handled the same flow correctly 19 times.
Run 20 interpreted “next Friday” differently, used the wrong timezone and submitted the booking before repeating the final date back to the user.
Conversation sounded completely normal.
Tool call was valid.
Booking succeeded.
Wrong day.
If you score that as 19/20, the agent is 95% successful.
Sounds decent.
But the five failures below cannot all have the same weight:
slightly awkward wording
asked the same question twice
took too long to respond
suggested the wrong date but corrected it
actually submitted the wrong booking
The last one should probably block release by itself.
My current approach is:
1. Check the tool payload deterministically
Exact date, timezone, account, duration and resource.
Not “did the response sound correct?”
2. Repeat ambiguous scenarios
“Next Friday”, “tomorrow evening”, “after lunch”, “same time next week”.
Run them multiple times because one clean result means very little.
3. Require confirmation before action
The agent should repeat the absolute date and timezone before submitting anything expensive or annoying to reverse.
4. Compare every model or prompt update
A change can improve conversation quality while quietly making date parsing worse.
I’ve been using TestMu Agent Testing for this layer because it generates different personas/scenarios and lets us compare the actual agent behaviour across releases, including action-level failures rather than only grading the final sentence.
Hamming and Cekura are also worth looking at for voice-agent QA. Maxim is broader on evaluation and observability.
Different emphasis, same basic lesson:
A nice average score can hide one very expensive failure.
No test platform will guarantee it finds every rare edge case.
But evaluation should at least reflect business damage instead of pretending every failed run is equal.
Do you optimise agent evals for average success, worst-case failure or expected business damage?