Skip to content
Voice AI

How to Build an AI Voice Agent in 2026: The Parts, the Choices and the Order to Build In

A plain guide to building an AI voice agent for phone calls: the six parts every agent needs, the cascade and speech-to-speech designs, what to choose for telephony, speech recognition, the language model and the voice, the build order that avoids rework, and what it costs to run, with list prices read on November 12, 2026.

Jaffar Jawed · Co-Founder
Published
9 min read
  • Voice AI
  • Technical Guide
  • Build
  • Telephony
In short

Quick answer

To build an AI voice agent you connect six parts: telephony, streaming speech recognition, a language model, a voice, turn-taking logic and tools such as calendar booking. Build one call flow on a real phone number first, then tune turn-taking, then add tools, transfers and failure handling. Running costs are about 7 to 31 cents a minute on developer platforms. A finished AI voice agent skips the build.

How do you build an AI voice agent?

An AI voice agent for phone calls has six parts: a phone connection, speech recognition, a language model, a voice, turn-taking logic that decides when to listen and speak, and tools that let it act, such as booking a calendar slot. You can assemble them yourself on a framework, configure them on a developer platform, or buy a finished agent. Building a demo takes days; building one that handles interruptions, noise, transfers and failures on a real phone line takes months. This guide covers the parts, the choices and the order. If you want the result without the build, Botcadence's AI voice agent is a finished one.

Key takeaways

  • Six parts: telephony, speech recognition, language model, voice, turn-taking and tools.
  • Two designs: a cascade of separate speech, model and voice services, or a single speech-to-speech model. Cascades are easier to control and debug; speech-to-speech can respond faster.
  • Three routes: a framework you host, a developer platform, or a finished product.
  • Build in this order: one call flow on a real phone number, then turn-taking, then tools, then transfers and failure handling, then monitoring.
  • Running costs on list prices are about 7 to 31 cents a minute on developer platforms, before your own time.

The six parts

PartWhat it doesExamples
TelephonyConnects a phone number to your software over SIPTwilio, Telnyx, Plivo, or any SIP trunk
Speech recognitionTurns the caller's audio into text, streamingDeepgram, AssemblyAI, Google, OpenAI
Language modelDecides what to say and which tool to callModels from OpenAI, Anthropic, Google and others
VoiceTurns the reply into audio, streamingElevenLabs, Cartesia, Deepgram, Google
Turn-takingDetects speech, decides when the caller has finished, handles interruptionsVoice activity detection plus a turn-detection model
ToolsLets the agent book, look up, transfer and recordCalendar and CRM APIs, your own endpoints

Around these sit the parts nobody demos: noise and echo handling, voicemail detection for outbound calls, call recording, transcripts, retries when a provider fails, and a way to see what went wrong.

Cascade or speech-to-speech

Cascade (speech to text, model, text to speech)Speech-to-speech model
How it worksThree services in sequenceOne model takes audio in and produces audio out
ControlYou see and can edit the text at each stepLess visibility into what it heard
Tool calls and business rulesMature and predictableImproving
SpeedEach stage adds timeFewer stages
Choice of voiceAny voice providerThe model's own voices
CostSum of three servicesPriced on audio tokens, often higher
DebuggingRead the transcript at each stageHarder

Most business phone agents today are cascades, because a transcript you can read, a model you can swap and rules you can enforce matter more than the last few hundred milliseconds. Speech-to-speech is worth testing when naturalness is the product.

Three routes to a working agent

RouteWhat you doWhat you getFits
Framework you hostWrite the agent in code on an open framework such as LiveKit Agents or Pipecat, and run the serversFull control of every partEngineering teams with real-time experience
Developer platformConfigure the agent and write prompts and tools on a hosted platform such as Retell AI, Vapi or Bland AIThe plumbing is done; you own the logicDevelopers who want to ship in weeks
Finished productGive your information and rules to a product built for a job, such as an AI receptionistA working agent with booking, transfers and recordsBusinesses without developers

Build vs buy an AI voice agent goes through the decision.

The build order

  1. One call flow on a real number. Buy a number, point it at your agent over SIP and make it handle a single task end to end, such as booking an appointment. Test from a mobile phone, not a browser.
  2. Turn-taking. Tune when the agent decides the caller has finished, and make it stop speaking when interrupted. This step decides whether the agent feels natural. Phone audio, background noise and echo make it harder than it looks.
  3. Grounding. Give the agent your real information and make it say when it doesn't know. An agent that invents a price is worse than no agent.
  4. Tools. Connect the calendar, the CRM and any lookup. Make every tool call safe to retry, and decide what the agent says while it waits.
  5. Transfers. Add handoff to a person, with a spoken summary, and a fallback for when nobody answers.
  6. Failure handling. Decide what happens when speech recognition, the model, the voice or a tool is slow or down: a retry, a simpler reply, a message taken.
  7. Outbound specifics, if you need them: voicemail detection, calling hours, retries, opt-outs and the consent rules in are AI phone calls legal?.
  8. Monitoring. Record per-turn timings, transcripts and outcomes, and read real calls every week.
  9. Load. Test several calls together and find out what happens at your limit.

Teams that start with step 4 or 5 usually rebuild, because turn-taking changes how every later step behaves.

The choices that matter most

  • Speech recognition: accuracy on names, addresses and numbers over the phone, in your callers' accents and languages. Test with recordings of real calls.
  • Language model: the smallest model that follows your rules and calls tools reliably. Larger models are slower and cost more per minute.
  • Voice: one that is clear on a phone line at speed, in each language you need.
  • Turn-taking: a model-based turn detector is better than a silence timer for callers who pause to think.
  • Telephony: a carrier with numbers where your callers are, and SIP support for transfers.

What it costs to run

List prices read on November 12, 2026:

PartList price
TelephonyTwilio inbound local $0.0085 a minute; outbound $0.014; a local number $1.15 a month
Speech recognitionDeepgram Nova-3 streaming $0.0077 a minute
VoiceElevenLabs Flash $0.04 per 1,000 characters; Deepgram Aura-2 $0.03
Language modelPer token; as a per-minute guide, Retell AI lists models from $0.0016 to $0.32 a minute
Platform, if you use oneVapi $0.05 a minute; Retell AI $0.055
Voicemail detection$0.0075 a call on Twilio

On a developer platform the total is about 7 to 31 cents a minute. Self-hosting removes the platform fee and adds servers and the engineers who keep them running. AI voice agent cost per minute works through the sums.

What is hard in production

  • Noise and echo that make the agent hear itself or the television.
  • Callers who pause, restart and talk over it.
  • Names, addresses, emails and numbers, which need reading back.
  • Tools that time out in the middle of a call.
  • Transfers across carriers, and what the caller hears while waiting.
  • Voicemail and phone menus on outbound calls.
  • Regulation: consent for outbound AI calls, recording consent by state, and sector rules such as HIPAA.
  • Change. Every model or prompt update can break something that worked; you need test calls you can repeat. See how to test an AI voice agent.

Where Botcadence fits

Botcadence is the third route: a finished agent. It connects to numbers in your own Twilio, Telnyx or Plivo account or on a SIP trunk, lets you choose among several speech, model and voice providers, includes model-based turn detection, interruption handling and noise reduction, and comes with calendar booking, warm transfer, on-call alerts, outbound campaigns and call records. It is $299 a month for 1,500 minutes and starts each account at one concurrent call. Developers who want to write their own agent logic will be happier on a framework or a developer platform; our LiveKit telephony noise and echo cancellation guide covers one of the harder parts. The AI receptionist for small business page shows the finished product.

Questions readers ask

How to Build an AI Voice Agent FAQ

Six parts: a phone connection over SIP, streaming speech recognition, a language model, a streaming voice, turn-taking logic that handles pauses and interruptions, and tools for actions such as booking. You also need recording, transcripts, failure handling and monitoring.

Partly. Developer platforms offer visual flow builders, and finished products are configured in a dashboard with no code. Custom logic, your own integrations and unusual call flows still need a developer on most platforms.

A demo that books an appointment can be built in days on a developer platform. A production agent that handles noise, interruptions, transfers, tool failures and compliance on a real phone line usually takes months and continuing maintenance.

Most business phone agents use separate services in a cascade, because the text at each step can be read, checked and controlled, and tool calling is more predictable. Speech-to-speech models have fewer stages and can respond faster; test both on your own calls.

Turn-taking on real phone audio: deciding when the caller has finished, stopping when interrupted, and not reacting to background noise or the agent's own echo. After that, reliable tool calls and transfers that never leave a caller stranded.

Sources

  1. Twilio voice pricing, United States (checked November 12, 2026)
  2. Deepgram pricing (checked November 12, 2026)
  3. ElevenLabs API pricing (checked November 12, 2026)
  4. Retell AI pricing (checked November 12, 2026)
  5. Vapi pricing (checked November 12, 2026)
  6. LiveKit Docs: Turn detector (checked November 12, 2026)
  7. LiveKit Docs: Answering machine detection (checked November 12, 2026)

Written by

· Co-Founder, Botcadence

Jaffar Jawed co-founded Botcadence in 2025 and leads product and go-to-market. He writes about AI voice agents, inbound answering workflows, lead qualification and the telephony stack behind real-time voice, including the company's open-source work on LiveKit noise and echo cancellation.

Get Started Today

See These Strategies in Action

Book a free demo and we will build a working AI agent for the channel this article covers, tailored to your business goals. No commitment required.

  • Voice AI plan $299 a month
  • Managed voice live in about 5 business days
  • Works with your existing tools

Ask AI about this page