语音 AI 基础设施：2026 年的技术栈解析

Voice AI infrastructure is the set of models, connections, tools, and services that lets an AI agent listen to a call, decide what to do, take action in other systems, and answer out loud. And, with leading providers like ElevenLabs, all of this happens in a fraction of a second.

A production voice agent runs on a stack of layers that includes speech recognition, turn-taking, a large language model, orchestration, tools, knowledge retrieval, speech synthesis, telephony, and monitoring.

This guide explains the AI agent architecture behind a working voice agent. Each section converts what a layer does, why it matters in the wider infrastructure, and how ElevenAgents implements it, with Eleven v4 as the example voice layer.

How voice AI infrastructure fits together

A voice agent turns audio into text, reasons based on that input, generates an answer, and then turns that response back into audio that the customer hears. That cascade of actions is made up by a robust stack of AI agent architecture.

Here are the main layers that contribute to voice AI infrastructure in 2026:

1. Speech recognition: Automatic Speech Recognition (ASR) technology converts the caller's audio into text for processing.
2. Turn taking: Turn taking decides when the caller has finished and when to stop talking, allowing for more fluid, responsive conversation.
3. Language model(s): Interprets the transcribed request and generates a reply, after pulling necessary context.
4. Orchestration: An AI orchestration layer routes the conversation between prompts, subagents, tools, and humans.
5. Tools and MCP: Connect the agent to external systems, such as CRM platforms or contact centers, to perform tasks.
6. Knowledge and memory: Supply facts and caller context via Retrieval Augmented Generation.
7. Speech synthesis: Uses Text to Speech to transform the text generation reply into audio.
8. Transport and telephony: Carries audio between the agent and the caller.
9. Guardrails, testing, and monitoring: Keeps the agent on task and measurable.

Together, these components work to create real-time conversations between AI agents and calling customers.

Voice AI architecture diagram

The flow isn’t a straight line. ElevenAgents starts speaking after it receives “enough words and a comma” from the language model, meaning that audio begins playing before the full reply exists. Toll calls and retrieval run in the middle of a turn. Interruptions can arrive and are handled naturally by the agent while it generates speech. 

Two ways to build voice AI architecture 

ElevenAgents supports two voice modes. In Custom mode, you pick the language model, the Text to Speech model, and the speech recognition settings yourself.

In Realtime mode, you select an Eleven Realtime model and ElevenLabs manages those settings for you. Eleven Realtime v1 Mini is the default and has the lowest latency and cost. Eleven Realtime v1 Max reasons before it responds, which suits more complex instructions and cases where multi-step tool use is required.

The system prompt, chosen voice, tools, attached knowledge base, and turn settings work the same in both modes. Custom mode is the better choice when you need a specific language model, a custom LLM, or a specific TTS model (for performance or latency optimization.)

Speech recognition: Turning audio into text

Speech recognition, also called ASR or speech to text, is the first layer of the stack.

What it does

The model transcribes the caller's audio continuously while they speak. Doing so continuously allows the rest of the pipeline to start working before the sentence ends, saving on latency. Advanced models will handle everything from thick accents and different speaking speeds to background noise.

Why it matters for voice AI infrastructure

Transcription is arguably one of the most important layers in the stack because any errors here will permeate through the rest of the stages. Mishear a booking reference? Wrong customer record actioned. Get a medication’s name wrong? Potential disaster. 

In its Trust Framework for Healthcare, ElevenLabs shows how high the stakes get with one misheard syllable: hydroxyzine versus hydralazine.

How ElevenAgents does it

ElevenAgents uses Scribe v2 Realtime, which transcribes speech in approximately 150 ms. ElevenLabs trained it for noisy backgrounds, diverse accents, and identifiers such as names, emails, and IDs. On the FLEURS multilingual benchmark across 30 languages, ElevenLabs reported the lowest word error rate of any low-latency ASR model. 

You enable it under the Advanced section of the agent configuration. In the same white paper mentioned above, ElevenLabs reports that Scribe v2 Medical leads every model on the public Eka Medical ASR benchmark at a 7.0% word error rate as of publication.

Accurate (and rapid) transcription gives the next layers the clean input they need without adding significant latency.

Turn taking: Knowing when to speak and when to stop

Turn taking is the set of rules that decides when the agent replies and when it listens. Being able to effectively understand turns in conversation and act upon them is what separates a conversation from a legacy IVR phone menu.

What it does

There are a few functions that a turn taking component provides to voice AI infrastructure:

• Voice activity detection (VAD): VAD classifies incoming audio as speech or silence, dozens of times per second.
• End-of-turn detection: Judges whether the caller has finished, using tone, rhythm, and word context.
• Barge-in: Lets the caller interrupt, at which point the agent stops talking to allow the user to continue.
• Backchannels: Short acknowledgments such as "yeah" that do not take the turn.

Why it matters

A gap of about 200 ms is the typical gap between speakers in a human conversation. An agent that answers too early cuts callers off, while one that waits too long creates silences that make the call feel broken or unresponsive. 

For example, consider a caller who says "I'd like to book Friday afternoon, but..." and pauses. A silence timer replies at the pause and misses the qualification. A model that reads the words and the rhythm waits, allowing the user to finish, "It can’t be between 3 and 5 p.m., as I’ll be busy.”

One agent misqualifies the caller’s intention and schedules the meeting at the wrong time. The other, with effective turn taking abilities, makes sure to avoid the specified time period. 

How ElevenAgents does it

ElevenAgents runs a proprietary turn-taking model that analyzes tone, rhythm, and word context instead of relying on silence timers alone. When a caller interrupts, the signal that stops the agent lands in under 200 ms. 

ElevenLabs measures a median of ~150 ms from request to audible speech with Eleven v4 Turbo on ElevenAgents. 

The behavior is configurable via Conversation flow:

Setting

What it controls

Turn eagerness

Eager responds at the earliest opportunity. Normal is the balanced default. Patient waits longer and suits phone numbers, addresses, and emails.

Speculative turn

Starts the language model reply before the caller is confirmed finished. It cuts latency and can raise LLM cost.

Interruptions

Turn them off when every word must be heard, such as legal disclaimers and safety instructions.

Ignore list

Words such as "right" that should not interrupt the agent.

Skip turn tool

Keeps the agent quiet when the caller says "give me a minute".

Turn timeout

Takes a turn after 1 to 30 seconds of silence. Use 5 to 10 seconds for casual conversations and 10 to 30 when callers need time to think.

Soft timeout

Plays filler speech such as "Hhmmmm...yeah." when the model is slow. Range 0.5 to 8 seconds, with 3 seconds recommended.

Noisy environment detection

An Enterprise setting that detects background voices and asks the caller to move somewhere quieter, at most once per conversation.

Workflows can change turn eagerness mid-call. Your client also receives a vad_score event, a 0 to 1 probability that the user is speaking. 

Well-tuned turn-taking decides whether callers experience the agent as attentive or abrupt. For more on the concepts, see our full guide on turn taking.

The language model: Reasoning and generating the reply

The large language model (LLM) is the reasoning layer. It reads the transcript, the instructions, and the conversation so far, then writes the response or decides to call a tool.

What it does

A scripted voice bot recognizes a handful of commands. An LLM interprets requests phrased many different ways, holds context across turns, and decides when to retrieve information or hand off.

Why it matters

Models differ in reasoning depth, speed, and cost. For a live voice conversation, response time matters as much as answer quality, so the model choice is a latency decision and a cost decision.

How ElevenAgents does it

ElevenAgents gives you one interface to models from Google, OpenAI, and Anthropic, plus open models hosted by ElevenLabs. You can also connect your own LLM by specifying an endpoint and storing credentials as secrets. ElevenLabs passes third-party LLM costs through at the provider's published rate with no markup.

You can choose the model per subagent. The Enterprise Conversational Agent Playbook, Volume 2 recommends reserving higher-performance models for complex reasoning and using cheaper, low-latency models for tasks such as intent classification and FAQs.

Providers have outages. LLM cascading handles them: if the primary model returns an API error, times out, or sends an empty response, ElevenAgents tries a backup sequence. The default timeout from primary to backup is 4 seconds, adjustable from 2 to 15 seconds, and the system makes at least 3 attempts across the available models. 

A custom LLM used as the primary model retries itself at least 3 times, unless you define your own backup order. The right model per task keeps the agent both fast and affordable.

Orchestration: Routing work between agents and humans

Agent orchestration coordinates the prompts, subagents, tools, and people involved in finishing a task. It decides what happens next at every point in the call.

What it does

One agent with one long prompt grows hard to manage as responsibilities pile up. Orchestration splits the work into stages, each with its own instructions, model, and tools, and moves the caller between them.

Why it matters

Smaller prompts are easier to test and cheaper to run, because each interaction loads less text. Splitting a flow lets a cheap model handle triage and FAQs and hands off to a stronger model only when a request needs it.

How ElevenAgents does it

Workflows in ElevenAgents are a visual builder for branching conversation graphs. 

The nodes are:

• Subagent: Changes or overrides the base agent, including its prompt, LLM, voice, tools, and knowledge base.
• Say: Speaks one message, either literal text or an LLM-generated line.
• Dispatch tool: A dedicated step that guarantees a tool call, with separate success and failure paths.
• Update state: Assigns dynamic variables as the flow passes through.
• Agent transfer: Hands the call to a different agent.
• Transfer to number: Moves the call from the AI agent to a human over the phone system.
• End call: Closes the conversation.

Edges between nodes carry natural-language conditions that the LLM evaluates in real time. A support flow might start with a routing subagent on a fast model, send billing questions to a billing subagent on a stronger model, and finish with a transfer-to-number node when the caller asks for a person. After launch, the Workflow tab in analytics shows entries, average time spent, terminations, and edge distribution for each node.

Orchestration turns a single agent into a system you can design, measure, and improve one node at a time.

Tools and MCP: Letting the agent act

Function calling, also called tool calling, lets an agent request actions from external software that you have connected to your tech stack. 

It turns a question-answering interface into an agentic AI system that is able to complete work and interact with other tools on your behalf.

What it does

Take a caller who asks to move tomorrow's appointment to Friday. The agent retrieves the booking, checks Friday's availability, updates the scheduling system, and confirms the change. Each step is a tool call that takes action.

Why it matters

Tools are where conversations produce outcomes. Each tool needs clear inputs and outputs, and the agent needs to know whether a call succeeded before it tells the caller it did.

How ElevenAgents does it

ElevenAgents supports client tools that run in the user's application, webhook tools that call external APIs, code tools that run custom JavaScript in a sandbox on ElevenLabs infrastructure, MCP tools, and built-in system tools. 

The system tools include end call, language detection, agent transfer, transfer to number, skip turn, play keypad touch tone, and voicemail detection. You can add ambient audio while a tool runs and control whether callers can interrupt during execution.

The Model Context Protocol (MCP) is an open standard for how applications give context and tools to language models. ElevenAgents connects to MCP servers over SSE and streamable HTTP. Workspace admins opt in first, since MCP is disabled by default for every workspace. You then choose an approval mode for each server: Always Ask, Fine-Grained Tool Approval, or No Approval. 

Client events such as client_tool_call and mcp_tool_call report tool activity, and the MCP event reports four states: loading, awaiting approval, success, and failure. Reliable tools give the agent real-world reach, and clear approval rules keep that reach under control.

Knowledge and memory: Facts and caller context

Retrieval-augmented generation (RAG) gives the agent facts from your own content, such as an internal fact database or wider client history log. Memory and personalization give it context about the caller that allows your agent to improve personalization and utility for your business.

What it does

A language model has a training cutoff and no access to your documentation or records. RAG searches a knowledge base for each query, retrieves the most relevant passages, and supplies them to the model to give it the extra context needed for a useful response. 

Personalization data tells the agent who is calling and what is relevant to them.

Why it matters

Grounding replies in approved content keeps answers accurate. Retrieval speed matters too, as every millisecond it adds is another that your customers will be waiting for. Personalization functions also stop callers from having to repeat information across different calls, as your agents will have the CRM records on hand at the start of each new session.

How ElevenAgents does it

You add documents to the knowledge base as files (PDF, DOCX, TXT, MD, HTML, or EPUB, up to 20 MB each), URLs, or pasted text. A document uses full context when its extracted text fits in the prompt, up to roughly 300,000 characters. Larger documents use RAG. Retrieval can run every turn or on demand, and the default usage mode is Auto, which uses RAG when it is enabled and the document is indexed.

ElevenLabs works on retrieval speed directly. Its engineering article, How we reduced RAG latency, documents a drop in median RAG latency from 326 ms to 155 ms using models self-hosted on infrastructure ElevenLabs already operates.

For caller context, ElevenAgents uses dynamic variables written as {{ var_name }}, which fill messages, system prompts, and tools. For inbound calls over Twilio, SIP trunk, WhatsApp, and SMS, ElevenAgents can fetch initiation data from your server before the conversation starts. Your server looks up the caller by caller_id and returns the variables, for example, from your CRM, with a response of up to 256 KB. After the call, post-call webhooks deliver evaluation results and extracted data to your external systems.

Together, retrieval and personalization give the agent both the right facts and the right caller.

Speech synthesis: Giving the agent a voice

Text to speech (TTS) turns the language model's reply into spoken audio. It is the layer the caller actually hears.

What it does

The model converts text into audio with natural pacing and emotional delivery. In a live call it streams audio in chunks so playback starts before the full reply is generated.

Why it matters

Latency and delivery both shape how callers rate the call. Two numbers matter for real-time work: the time to generate audio and the time before the caller hears the first sound. Names, drug names, and technical terms also need correct pronunciation, or callers lose confidence in the agent.

How Eleven v4 powers the voice layer

Eleven v4 is ElevenLabs' most emotive Text to Speech model yet, built on an entirely new architecture. Eleven v4 Turbo is the low-latency variant for real-time use, such as agents, optimized with ElevenAgents as one system.

Eleven v4 Turbo

Figure

Median inference latency

About 100 ms

Median time to first speech

About 150 ms

Languages

90+

ElevenLabs measured the latency figures in September 2026 over WebSocket streaming with network latency removed. Eleven v4 ranked first on the Artificial Analysis Provider Voice Arena Leaderboard in September 2026*, and about 75% of listeners preferred it in blind tests**.

* Artificial Analysis, Provider Voice Arena Leaderboard, Sept 2026

** Based on head-to-head user preference testing against Cartesia Sonic 3.6, Inworld TTS-2, Google Gemini 3.8 Flash-Lite TTS, and Google Gemini 3.8 Flash TTS.

Eleven v4 supports inline audio tags such as [laughs] for delivery and IPA phoneme support for pronunciation. In healthcare agents, that pronunciation control covers medical terms. ElevenAgents adds agent-level controls on top. Multi-voice support lets one agent switch between voices for multi-character conversations. Speed ranges from 0.7x to 1.2x. Each additional language can use its own voice. Pronunciation dictionaries take IPA or CMU notation for specific words. 

A fast first sound and an accurate, expressive voice make the agent feel like a conversation partner.

Telephony and transport: Connecting the agent to callers

The transport layer carries audio between the caller and the agent. Depending on which telephony channel you use, the integrations you’ll include in your voice AI infrastructure here will change.

What it does

Phone calls travel over a telephone network and SIP, while browser and app conversations travel over WebRTC or WebSocket. A website widget wraps the browser connection in a ready-made interface.

Why it matters

Each channel has different setup needs and different audio constraints, so decide your channels early. A phone-first deployment needs trunking and number management. A product deployment needs an SDK.

How ElevenAgents does it

• SIP trunking: Connects your phone system directly to ElevenLabs, inbound and outbound. It works with Twilio, Vonage, Sinch, Infobip, Telnyx, and other standard providers. Signaling runs over TCP on port 5060 or TLS 1.2 or higher on port 5061, and audio uses G.711 or G.722. Plan for at least 100 Kbps per call. Concurrent call limits follow your subscription tier, and callers wait when call queueing is enabled.
• Native Twilio integration: Import a purchased Twilio number for inbound and outbound calls. Verified caller IDs work for outbound only.
• Batch calls: Upload a CSV or XLS recipient list, and each extra column becomes a dynamic variable. You can run immediately or schedule a start. Batches use the lower of 50% of your workspace concurrency limit or 70% of your agent's concurrency limit, which leaves capacity for incoming calls.
• WebRTC and WebSocket: The SDKs use WebRTC for voice and WebSocket for text-only conversations by default. Private agents authenticate with a signed URL or a conversation token created on your server. The WebSocket endpoint is wss://api.elevenlabs.io/v1/convai/conversation.
• SDKs: Python, JavaScript, React, React Native, Swift, Kotlin, and Unity.
• Widget: Add the <elevenlabs-convai> element to a page. It supports voice only, voice plus text, and chat modes, and the agent can show quick replies and carousels.

One agent can serve phone, web, WhatsApp, SMS, and email, which lets teams build once and deploy across channels.

Guardrails: Keeping the agent on task

Guardrails are controls that keep an agent's behavior inside defined boundaries. They work alongside grounding, as a guardrail limits what an agent says without making the facts behind it correct. Guardrails run in parallel to agent conversations, helping to keep your AI agent conversations compliant and accurate.

What it does

ElevenAgents applies guardrails at three levels. The system prompt sets allowed and disallowed behavior, ideally under a # Guardrails heading. User input validation screens messages for prompt injection. Response validation checks each reply before delivery.

Why it matters

An agent that wanders off topic or invents a policy erodes trust fast. A customer support agent should retrieve the refund policy from an approved source before answering, and a guardrail enforces that boundary.

How ElevenAgents does it

ElevenAgents introduced Guardrails 2.0 in September 2026. With this new system, here’s a glimpse into the configurations that help guardrails to work effectively:

• Focus: Keeps the agent on-topic and aligned with its system prompt.
• Manipulation: Detects prompt injection and ends conversations that pose a risk.
• Content: Flags categories such as sexually explicit, violent, or politically sensitive material.
• Custom: Business rules you write in natural language, evaluated on every response.

Execution mode decides the latency trade-off. Streaming mode lets the reply begin before the check completes, so under 500 ms of audio can play before a block ends the call, and ElevenLabs recommends it for voice agents. Blocking mode waits for validation and adds roughly 200 to 500 ms. 

When a custom guardrail triggers, the trigger action either ends the call or retries the response up to three times with feedback you write. Your client can listen for the guardrail_triggered event.

Layered guardrails with a chosen latency profile keep behavior predictable without slowing every reply.

Testing and experiments: Proving the agent works

Testing checks an agent before launch. Experiments check changes against live traffic after launch.

What it does

ElevenAgents has three agent test types. Simulation runs a full conversation against a simulated user and checks whether it reaches a defined outcome, with up to 50 turns (default 5). Next Reply tests judge only the agent's next message against your criteria. Tool Call tests verify that the agent uses the right tool with the right parameters, matched by exact value, regex, or LLM evaluation. In simulations you can mock no tools, all tools, or selected ones.

Why it matters

A single passing run shows an agent can succeed. Running tests repeatedly shows how often it will. The API accepts repeat_count from 2 to 20, and results show a pass rate. Green means 100%, amber means 80% or higher, and red means below 80%. Failures are grouped by reason. Prompts, models, workflows, and integrations all change behavior, so testing continues after launch.

How experiments work

Experiments are A/B tests built on agent versioning. You branch your configuration, change one variable such as the system prompt, workflow, voice, tools, knowledge base, or LLM settings, and route a share of live traffic to the variant. ElevenLabs recommends starting at 5 to 10%. 

Routing is deterministic by conversation ID, so a given user stays on the same branch, and percentages must total 100%. CARS24, an ElevenLabs customer, ramped traffic through 5%, 10%, 20%, 50%, and 100% with a two-day hold at each step, as described in the report.

Pre-launch tests and live experiments together turn changes into measured improvements.

Observability: Measuring what happens in production

Observability is the ability to see what the agent did, how it performed, and, if the case arises, where it failed. An agent that works in a test can behave differently with real callers, which makes full observability and auditable record keeping a must.

What it does

Conversation analysis scores every call. Success evaluation applies criteria you define for goal achievement and quality, and data collection extracts structured fields such as contact details and issue type. The analytics dashboard adds call volume, duration, cost, error rates, and latency. Active calls update in real time, and alerts fire when error rates spike.

Why it matters

Without monitoring, a poor call is an anecdote. With it, you can see which node, tool, or model caused the problem.

KPI

What it tells you

Where ElevenAgents surfaces it

Agent response latency

How long callers wait for a reply

Analytics dashboard, median and percentiles

Turn-taking latency

How quickly the agent takes its turn

Analytics dashboard, p50, p90, and p99

Error rate

Share of conversations with errors

Dashboard, broken down by tool, LLM, and connection errors

Success rate

Goal achievement per criterion

Evaluation criteria, with success, failure, and unknown rates

Containment rate and handling time

Calls resolved without a human, and call duration

Experiment comparisons

Cost per resolution

What each resolved conversation costs

Experiment comparisons

Pick the metrics that are most useful for the specific job you have in mind. A support agent leans on resolution and escalation, while an outbound agent would mainly focus on connection and completion.

Scale, reliability, and human handoff

Voice AI infrastructure needs to be able to deliver whether you’re talking to 5 customers or 500,000. Even when working at volume, these components need to survive and be able to escalate to humans when needed.

What it does

Scale means supporting many simultaneous conversations across speech recognition, LLM inference, speech synthesis, integrations, and telephony. Reliability means each of those dependencies has a defined failure path. Handoff means a clear route to a human.

While not strictly a component within voice AI infrastructure, they’re all essential parts of what makes a good system in practice. 

Why it matters

A system that handles ten test calls can behave differently at ten thousand. A booking tool that times out must not lead the agent to confirm a booking that never happened. A caller who asks for a person must reach one with context intact, typically via warm transfer.

How ElevenAgents does it

Concurrency limits follow subscription tier, and call queueing holds callers when a limit is reached. LLM cascading covers model outages. Transfer-to-number nodes, the transfer-to-number system tool, and agent transfer cover handoff, and evaluation criteria can decide when to escalate. Silent periods are billed at 5% of the usual per-minute rate.

ElevenLabs reports production results at scale. ElevenAgents powers more than 10 million conversations per week. Meesho automates about 90% of support queries across 60,000+ calls per day, per the report Building Enterprise-grade Voice Agents. In the Czech Republic, ElevenLabs-powered agents handle around 5,000 calls per day on national benefits hotlines and independently resolve 85% of inquiries, per the State of Conversational Agents in Government report.

Planning for load, failure, and handoff from the start is what separates a demo from a deployment.

What to look for in production-ready voice AI infrastructure in 2026

Use this checklist to evaluate any voice AI platform against your requirements.

• Speech recognition: accurate transcripts for your languages, accents, and audio quality, including names and identifiers.
• Turn-taking: configurable end-of-turn behavior, barge-in, and a measured interruption response time.
• Model choice: a selectable LLM per agent or subagent, a custom LLM option, and automatic fallback.
• Orchestration: multi-step workflows and routing between specialized agents.
• Tools: webhooks, client tools, MCP support, and approval controls.
• Knowledge: your own documents, with retrieval latency you can measure.
• Personalization: caller data available before the first word is spoken.
• Voice: low time to first speech and control over pronunciation and expression.
• Channels: phone, web, mobile, and messaging, with SDKs and an API.
• Guardrails: configurable behavior limits with a known latency cost.
• Testing: simulated conversations, tool call checks, and repeat runs.
• Experiments and analytics: A/B tests, percentile latency, and error breakdowns.
• Scale and handoff: documented concurrency, queueing, and escalation to humans.

Leading AI voice agent systems like ElevenAgents offer you each of these components in one platform.

Get started today by reaching out to the team or build your own AI agent systems by signing up for an account.

Frequently asked questions about voice AI infrastructure