How do you build an AI voice agent?
An AI voice agent for phone calls has six parts: a phone connection, speech recognition, a language model, a voice, turn-taking logic that decides when to listen and speak, and tools that let it act, such as booking a calendar slot. You can assemble them yourself on a framework, configure them on a developer platform, or buy a finished agent. Building a demo takes days; building one that handles interruptions, noise, transfers and failures on a real phone line takes months. This guide covers the parts, the choices and the order. If you want the result without the build, Botcadence's AI voice agent is a finished one.
Key takeaways
- Six parts: telephony, speech recognition, language model, voice, turn-taking and tools.
- Two designs: a cascade of separate speech, model and voice services, or a single speech-to-speech model. Cascades are easier to control and debug; speech-to-speech can respond faster.
- Three routes: a framework you host, a developer platform, or a finished product.
- Build in this order: one call flow on a real phone number, then turn-taking, then tools, then transfers and failure handling, then monitoring.
- Running costs on list prices are about 7 to 31 cents a minute on developer platforms, before your own time.
The six parts
| Part | What it does | Examples |
|---|
| Telephony | Connects a phone number to your software over SIP | Twilio, Telnyx, Plivo, or any SIP trunk |
| Speech recognition | Turns the caller's audio into text, streaming | Deepgram, AssemblyAI, Google, OpenAI |
| Language model | Decides what to say and which tool to call | Models from OpenAI, Anthropic, Google and others |
| Voice | Turns the reply into audio, streaming | ElevenLabs, Cartesia, Deepgram, Google |
| Turn-taking | Detects speech, decides when the caller has finished, handles interruptions | Voice activity detection plus a turn-detection model |
| Tools | Lets the agent book, look up, transfer and record | Calendar and CRM APIs, your own endpoints |
Around these sit the parts nobody demos: noise and echo handling, voicemail detection for outbound calls, call recording, transcripts, retries when a provider fails, and a way to see what went wrong.
Cascade or speech-to-speech
| Cascade (speech to text, model, text to speech) | Speech-to-speech model |
|---|
| How it works | Three services in sequence | One model takes audio in and produces audio out |
| Control | You see and can edit the text at each step | Less visibility into what it heard |
| Tool calls and business rules | Mature and predictable | Improving |
| Speed | Each stage adds time | Fewer stages |
| Choice of voice | Any voice provider | The model's own voices |
| Cost | Sum of three services | Priced on audio tokens, often higher |
| Debugging | Read the transcript at each stage | Harder |
Most business phone agents today are cascades, because a transcript you can read, a model you can swap and rules you can enforce matter more than the last few hundred milliseconds. Speech-to-speech is worth testing when naturalness is the product.
Three routes to a working agent
| Route | What you do | What you get | Fits |
|---|
| Framework you host | Write the agent in code on an open framework such as LiveKit Agents or Pipecat, and run the servers | Full control of every part | Engineering teams with real-time experience |
| Developer platform | Configure the agent and write prompts and tools on a hosted platform such as Retell AI, Vapi or Bland AI | The plumbing is done; you own the logic | Developers who want to ship in weeks |
| Finished product | Give your information and rules to a product built for a job, such as an AI receptionist | A working agent with booking, transfers and records | Businesses without developers |
Build vs buy an AI voice agent goes through the decision.
The build order
- One call flow on a real number. Buy a number, point it at your agent over SIP and make it handle a single task end to end, such as booking an appointment. Test from a mobile phone, not a browser.
- Turn-taking. Tune when the agent decides the caller has finished, and make it stop speaking when interrupted. This step decides whether the agent feels natural. Phone audio, background noise and echo make it harder than it looks.
- Grounding. Give the agent your real information and make it say when it doesn't know. An agent that invents a price is worse than no agent.
- Tools. Connect the calendar, the CRM and any lookup. Make every tool call safe to retry, and decide what the agent says while it waits.
- Transfers. Add handoff to a person, with a spoken summary, and a fallback for when nobody answers.
- Failure handling. Decide what happens when speech recognition, the model, the voice or a tool is slow or down: a retry, a simpler reply, a message taken.
- Outbound specifics, if you need them: voicemail detection, calling hours, retries, opt-outs and the consent rules in are AI phone calls legal?.
- Monitoring. Record per-turn timings, transcripts and outcomes, and read real calls every week.
- Load. Test several calls together and find out what happens at your limit.
Teams that start with step 4 or 5 usually rebuild, because turn-taking changes how every later step behaves.
The choices that matter most
- Speech recognition: accuracy on names, addresses and numbers over the phone, in your callers' accents and languages. Test with recordings of real calls.
- Language model: the smallest model that follows your rules and calls tools reliably. Larger models are slower and cost more per minute.
- Voice: one that is clear on a phone line at speed, in each language you need.
- Turn-taking: a model-based turn detector is better than a silence timer for callers who pause to think.
- Telephony: a carrier with numbers where your callers are, and SIP support for transfers.
What it costs to run
List prices read on November 12, 2026:
| Part | List price |
|---|
| Telephony | Twilio inbound local $0.0085 a minute; outbound $0.014; a local number $1.15 a month |
| Speech recognition | Deepgram Nova-3 streaming $0.0077 a minute |
| Voice | ElevenLabs Flash $0.04 per 1,000 characters; Deepgram Aura-2 $0.03 |
| Language model | Per token; as a per-minute guide, Retell AI lists models from $0.0016 to $0.32 a minute |
| Platform, if you use one | Vapi $0.05 a minute; Retell AI $0.055 |
| Voicemail detection | $0.0075 a call on Twilio |
On a developer platform the total is about 7 to 31 cents a minute. Self-hosting removes the platform fee and adds servers and the engineers who keep them running. AI voice agent cost per minute works through the sums.
What is hard in production
- Noise and echo that make the agent hear itself or the television.
- Callers who pause, restart and talk over it.
- Names, addresses, emails and numbers, which need reading back.
- Tools that time out in the middle of a call.
- Transfers across carriers, and what the caller hears while waiting.
- Voicemail and phone menus on outbound calls.
- Regulation: consent for outbound AI calls, recording consent by state, and sector rules such as HIPAA.
- Change. Every model or prompt update can break something that worked; you need test calls you can repeat. See how to test an AI voice agent.
Where Botcadence fits
Botcadence is the third route: a finished agent. It connects to numbers in your own Twilio, Telnyx or Plivo account or on a SIP trunk, lets you choose among several speech, model and voice providers, includes model-based turn detection, interruption handling and noise reduction, and comes with calendar booking, warm transfer, on-call alerts, outbound campaigns and call records. It is $299 a month for 1,500 minutes and starts each account at one concurrent call. Developers who want to write their own agent logic will be happier on a framework or a developer platform; our LiveKit telephony noise and echo cancellation guide covers one of the harder parts. The AI receptionist for small business page shows the finished product.