AI Orchestration

A voice agent is not one model. It is a speech recognizer, a language model, a retrieval step, a handful of tool calls and a speech synthesizer, each with its own latency and failure modes. Orchestration is the code that runs them in the right order, passes the outputs along, handles timeouts and decides what the customer hears while the slow parts finish. In a chat agent the chain is shorter (no audio) but the same layer manages retrieval, tool calls, memory and channel delivery.

How it works. On a call, audio streams into speech recognition, which emits partial and final transcripts. The orchestrator decides when the caller has finished a turn, sends the text with conversation history and retrieved documents to the language model, streams the model's reply into text to speech, and plays it back. If the model requests a tool (check availability, look up an order), the orchestrator runs it, feeds the result back and may play a short filler so the line is not silent. It also handles barge-in: when the caller speaks over the agent, playback stops and the new audio is processed. Every stage is timed and logged.

Why it matters for voice quality and cost. Response time is the sum of the stages, so orchestration is where latency is won. Running recognition, generation and synthesis as overlapping streams rather than sequential steps is the difference between a one-second pause and a three-second one. It is also where failures become graceful: a retrieval timeout should produce "let me check that" and a handoff, not a dropped call.

Example. A heating and cooling contractor's inbound line gets a call at 6 pm: "my AC stopped and it's 95 degrees in here, can someone come tonight?" Recognition finishes the sentence, the orchestrator sends it to the model with the service-area document, the model asks for the after-hours dispatch tool, the orchestrator runs it, finds a technician slot at 8 pm, and the agent offers it while the caller is still on the line. The transcript, the tool call and the booking are written to the job system together.

Orchestration vs agentic workflow. Orchestration is fixed plumbing: the same stages run for every call. An agentic workflow is the model's choice of which tools to call for this request. Orchestration executes what the workflow decides and keeps the audio flowing while it does.

What to measure. Time from end of caller speech to start of agent audio (the number callers feel), broken down by stage. Tool call duration and timeout rate. Barge-in response time. Share of turns where a filler phrase was needed. A stage that drifts upward over a week usually points at a provider or network change.

Botcadence voice agents target sub-second response times with natural barge-in; measured latency depends on model, telephony path and network conditions, as covered under sub-500ms latency.