How are you measuring whether your AI agents are actually worth what they cost?
I'm looking at teams that run several agents in production (support triage, lead research, internal ops, that kind of thing). Seeing the model bill is easy. What seems hard to see:
- which agent is driving which part of the bill
- how many runs actually succeeded vs. needed a human to fix them
- how much human review time each agent eats
- and so what one *successful* result really costs, per agent
From what I've seen so far it's usually a spreadsheet stitched together from provider dashboards and logs, and when finance asks "is this worth it?" the honest answer is "probably, for some of them."
For people running 5+ agents:
Do you track cost per successful outcome at all? How?
Have you ever downgraded a model or killed an agent because of numbers like this?
Who asks for this at your company: engineering, finance, or nobody yet?
Disclosure: I'm exploring building a tool for this, so I'm mostly trying to learn whether it's a real problem or just mine. I'll share a summary of what people say in a follow-up post.