I crashed four production AI agents mid-action. None of them recorded what already finished.
I kept hitting the same thing building agents, so I went and measured it against other people's code rather than arguing about it.
Four published agent codebases, each pinned at a commit, each killed partway through an action, then retried the way a supervisor or worker pool would. Nothing reimplemented. Where I had to substitute something, I used the project's own documented extension points. Counts come from a ledger the fake service fsyncs before it responds, so a process that dies the instant after acting still counts as having acted. The agent is never asked what it did.
What happened:
- Cua (computer-use): Kill it mid-checkout and the retry charges the card again. Both crash positions, every run. Measured on cua-agent 0.8.4 and again on 0.9.0, with identical results.
- Skyvern: Any process death during a step strands the task in
runningforever. Not failed, not errored, so nothing alerting on failure notices. The only way out is a new run, which orders again. - Corsair: Two replicas spend the same rotating OAuth refresh token. A crash leaves the stored credential broken, every run, so the user has to reconnect the integration.
- Inconvo: An interrupted answer costs 60% more in model calls because the checkpointer records the graph node but not the fifteen model calls underneath it.
Four different failures, one shared cause: none of them records which actions already completed. The retry is a fresh process with an empty history, so it does the work again.
The part I found most useful: a study of 25,930 agent episodes found that agents reported success in 90% of episodes where they had duplicated an effect. So you can't ask the agent, and anything graded on self-reported traces scores "duplicated and lied about it" as a pass.
One thing I won't claim to fix: if a process dies between the action landing and the record of it committing, it still repeats. The effect and the record live in different systems, and nothing closes that gap. It's also the row any benchmark reporting zero duplicates has quietly excluded by only crashing at already-committed boundaries.
Harnesses and raw ledgers: https://github.com/cellaflow/cellaflow-benchmarks
No API key needed. Every call goes to a local fake, so it costs nothing to run.
Disclosure: I build a runtime for this, so I'm not neutral. The measured arm is their code at a pinned commit.
What I'd actually like to know:
- If you run more than one agent against shared state, what actually stops two of them taking the same action? Every answer I've seen in the wild is a lock, a dedupe table, or nothing. I'd like to know whether there's a fourth.
- And if you're running agents that touch money or send things, how do you handle the retry today? Keys you pass in yourself, or do you just accept the occasional double?