those of you running voice agents in prod — what actually happens between editing a prompt and real callers hearing it?
We have two voice agents in production. One does outbound calls, the other basically runs day-to-day admin for a small business — their customers talk to it every day.
Embarrassing confession: every time I touch the prompt, my entire testing process is... I call it 5 times, listen, and if it sounds fine, I ship. That's it. That's the pipeline.
We did properly measure it once — sat down and counted, roughly 9 out of 10 voice requests did the right thing. Felt great for a day. But that was one manual count. No idea what the number is today.
So I'm curious what everyone else is actually doing:
- when you change a prompt, does it hit a staging agent or some test-call suite first, or straight to prod like us?
- how do you find out something broke? for us it's honestly "the client texts me"
- did anyone build real tooling for this, or is it duct tape everywhere?
I read a ton of launch posts but nobody ever writes about week 6. genuinely want to know if everyone's secretly winging it like we are, or if we're just behind.