u/akl773

I have a bot on the Cloud API that handles text. Message comes in, an LLM answers it with a few tools attached, reply goes back out. That part is stable. Now I want people to be able to call the same number and talk to it, and I am stuck on how to wire the audio.

Two shapes I can see. First is a pipeline. Take the call audio, run STT, hand the transcript to the same model that already answers the text messages, TTS the reply and stream it back. The brain already exists so I would mostly be wrapping it, but that is three hops of latency and I would be writing my own turn detection and barge in handling.

Second is putting a speech to speech model directly in the call. Grok's Voice Agent API does this, OpenAI Realtime does too, audio in and audio out with function calling built in. Much less of my code sits in the path that way. Cost is that my prompt and tools need porting, and I lose the transcript step where basically all my debugging happens today.

From the docs the Calling API gives WebRTC media by default with SIP available if you enable it, so reaching the stream looks fine on paper. Less clear to me is the 2000 conversations per 24h messaging limit before production calling unlocks. Sandbox numbers get around that for testing, which only proves the plumbing works.

If you have run either version with actual users on it, how bad is the latency in practice, and how often does the model end up talking over the caller. Mine is an India number if that changes anything.

I have a bot on the Cloud API that handles text. Message comes in, an LLM answers it with a few tools attached, reply goes back out. That part is stable. Now I want people to be able to call the same number and talk to it, and I am stuck on how to wire the audio.

Two shapes I can see. First is a pipeline. Take the call audio, run STT, hand the transcript to the same model that already answers the text messages, TTS the reply and stream it back. The brain already exists so I would mostly be wrapping it, but that is three hops of latency and I would be writing my own turn detection and barge in handling.

Second is putting a speech to speech model directly in the call. Grok's Voice Agent API does this, OpenAI Realtime does too, audio in and audio out with function calling built in. Much less of my code sits in the path that way. Cost is that my prompt and tools need porting, and I lose the transcript step where basically all my debugging happens today.

From the docs the Calling API gives WebRTC media by default with SIP available if you enable it, so reaching the stream looks fine on paper. Less clear to me is the 2000 conversations per 24h messaging limit before production calling unlocks. Sandbox numbers get around that for testing, which only proves the plumbing works.

If you have run either version with actual users on it, how bad is the latency in practice, and how often does the model end up talking over the caller. Mine is an India number if that changes anything.

I build automations and agent stuff for small businesses, mostly restaurants, service companies, a few software teams. The thing I keep running into has nothing to do with models or frameworks and I don't see it talked about much here.

The brief arrives as two lines on WhatsApp. Something like "we want it to take the order and send it to the kitchen, same as our staff do". Fine. So I ask what the staff do.

Ask three people and you get three answers. The owner describes the process he designed four years ago. The manager describes a version with about six exceptions layered on top. The person actually doing it every day describes a fourth thing that involves a notebook.

On one restaurant build I asked for the menu, which felt like the simplest possible request. Got a PDF. Then it turned out the same dish is priced differently at two outlets, one outlet stops serving half of it after 4pm, and two items are the same dish with different names depending on who typed it in. There was no single answer to what is on the menu, and that is before any agent gets involved.

You cannot automate a process that only exists as a habit. So most of the work isn't the agent at all. It is sitting with people and forcing the business to write its own rules down, usually for the first time, then getting someone senior to sign off on the version that will actually be encoded.

The part that annoys me is that clients don't want to pay for that. It doesn't look like software, it looks like meetings. But every project that went badly for me went badly there, not in the build, and the ones that went well were the ones where somebody with authority sat down and made the calls.

So genuinely curious how the rest of you handle it. Do you charge for that discovery as its own line item, or do you quote the build and quietly absorb it.

A decent chunk of my work this year has turned into the same job. Someone builds most of a product with an AI coding tool, gets to something that demos fine, then hits a wall and calls in a person.

Everyone expects the problem to be the code. Spaghetti, no tests, five different state patterns in one app. Honestly the code is usually okay. It's readable, it's consistent, it's better commented than most human handoffs I get.

What's broken every single time is the data model.

Last one I picked up had the same concept living in three tables under three names. Items, products and menu entries, all created at different points in the build, all half wired to each other. Price was stored on the item and also copied onto the order line, which sounds sensible until you realise nothing ever decided which one was true, so editing a menu quietly rewrote what old orders had cost. Nobody noticed for months because the screens all looked right.

That's the part that actually costs money. I can rewrite a component in an afternoon. I cannot un-record a year of orders against a schema that never made sense.

I think it happens because these tools never push back. You ask for a feature, you get a feature. They won't tell you the feature contradicts three tables you already have, because they can't see the business the tables are describing. A junior would at least have asked what a product is supposed to mean here, and got told, and written it down somewhere.

So when people say AI writes bad code I think they're aiming at the wrong thing. Code is the cheap part now, genuinely. The expensive part was always deciding what the data means, and that is the one step the whole workflow encourages you to skip.

Anyone else doing these cleanups seeing the same thing, or am I just getting a specific kind of client.

Built a thing that answers customer messages for small stores, mostly DMs and reviews. The whole pitch was that the owner never has to touch it.

First week of real users, almost every one of them asked for the same thing. Slow it down, let me see it before it goes out. One guy switched off auto send completely and just used the drafts.

Took me a while to accept they were not being paranoid. The messages it got wrong were never the normal ones, it was the refund threats and the angry review where a wrong reply costs a real customer. Owners can smell those in one line, the model cannot.

What fixed it was not better prompts, it was a parking rule. Anything with money in it, a complaint, or a name it has not seen before goes into a queue for the owner, everything else sends. Owner deals with 20 a day instead of 200 and nobody asks me to slow it down anymore.

So autonomy was never the feature, the sorting was. Anyone else building agents for non technical users end up in the same place, or did you find a way to get them comfortable with full auto.