BUILDING AN AI PA TO MAKE PHONE CALLS
Has anyone ever built one? For context, I'm building an AI PA for a client to make outbound calls on their behalf for day to day tasks. I'm using Vapi, ElevenLabs, Replit, and Twilio for the stack.
So far it can do basic stuff, Vapi recognizes the tools well like voicemail, IVR, holding, transferring etc. but the latency is 1000ms+. I tried ElevenLabs for the agent but it just wasn't recognizing voicemail or sitting on hold accurately, although the voices were much better.
It needs to be able to wait through IVRs, get context from ChatGPT/GrokBot where all the knowledge base lives mid call, wait through music, sit on hold, and recognize different voicemails. Issue is it keeps falling for false positives.
I'm wondering if anyone has built something like this before where it was super reliable for day to day use and what tools you used?