1.9M conversations across 150+ production agents. this is what a demo never shows you
we run 150+ sales agents in production. whatsapp, sms and instagram dms, real customers, real money. about 1.9m conversations through them at this point. one of them handled 50,000 in a single month during a client's launch.
every single one of those agents worked perfectly in testing. this is the stuff that only shows up once the volume is real. some learnings:
an agent with no stop condition will keep selling and talking to someone who already bought. this is the one that embarrasses you in front of a client. in production you need terminal state gates. when a "purchased", "booked" or "not qualified" flag flips, the agent stops, full stop, no matter how good the conversation reads. build this before you build anything clever.
the agent cant reliably report its own state. we spent weeks writing instructions like "set the booked flag to true when you schedule a call". doesn't work, and worse, it fails silently. the conversation looks perfect and the automation behind it never fires. what works is a separate evaluator that reads the transcript afterwards and sets the state. the conversational agent controls what it says. thats the whole surface. anything else you want it to do lives somewhere else.
follow ups are where the revenue is and where almost nobody builds. most leads don't answer the first message. the agent that books calls isn't the one with the best opener, it's the one that comes back on day two and day five without being annoying. one client's launch with 1k conversations, 47 of those conversations were recovering declined payments one at a time. that's not a conversational feat, its a workflow that fires reliably.
handoff to a human has to be a designed moment, not an escape hatch. "let me get someone to help you" with no routing behind it is worse than the agent just continuing. decide who gets pinged, on what channel, with what context attached.
the metric isn't response quality. this one cost us the most time by far. we spent months tuning conversations that read beautifully and booked nothing. booked calls (or sales) is the metric. once we started scoring on that, half of what we thought was good practice turned out to be the agent being pleasant instead of useful.
silent failures beat loud ones every time. a channel that disconnects but still shows "active" cost one client four days of leads. alerting on zero volume windows did more for retention than any feature we shipped that quarter.
none of this is about model quality btw. we changed models twice and it moved almost nothing. the machinery around the model is the product.
curious what other people hit, especially anyone running agents that touch payments or scheduling