Voice AI & Real-Time Multimodal

Why This Matters

AI-052 covers multimodal AI broadly: a model that can accept and produce images, audio, or text. Voice AI is a specific, much harder application of that capability, because a phone call or voice assistant has no tolerance for the multi-second round trips that are perfectly acceptable when someone is reading a chatbot's text response. Humans notice a pause longer than about 300 milliseconds in conversation; anything past roughly a second feels broken. Building a voice product means solving audio understanding and generation and a real-time systems problem at once, which is why it has become its own specialization rather than a footnote under multimodal AI.

Everyday Analogy

Reading a text message and having a phone conversation use the same language skills but feel completely different. You can pause for ten seconds before replying to a text with zero social cost; pause for ten seconds mid-phone-call and the other person asks "are you still there?" Voice AI has to hit the phone-call bar, not the text-message bar, even though under the hood it's still "understand input, generate output" like any multimodal system.

Speech-to-Speech vs. the Cascade Architecture

There are two architectural approaches to voice AI, and the difference explains most of the latency and naturalness gap between products:

• The cascade (older, still common). Audio in, run speech-to-text (transcription), pass the resulting text to an LLM, get a text response, run text-to-speech to synthesize audio out. Each stage is a separate model, each adds latency, and information is lost at each handoff: an LLM working from a transcript never hears tone, hesitation, or emotion in the original audio, and the text-to-speech stage never knows what the LLM was "thinking," only what it decided to say.
• Speech-to-speech (native, newer). A single model processes audio input and produces audio output directly, without transcribing to text as an intermediate step. This preserves paralinguistic information (tone, emphasis, pauses) that the cascade throws away, and it collapses three model calls into one, cutting the latency the cascade architecture accumulates at each handoff. OpenAI's Realtime API and comparable offerings from other labs use this architecture.

The cascade is simpler to build (you can mix and match best-in-class transcription, LLM, and voice-synthesis providers) and remains common in production. Native speech-to-speech is newer, harder to build, but structurally faster and more expressive.

The Real-Time Systems Problems

Beyond raw model quality, voice AI has to solve problems that text-based multimodal AI does not:

• Latency budget. The total round trip, audio capture, network, inference, audio synthesis, playback, needs to land under roughly 500 milliseconds to feel conversational. Every architectural choice (cascade vs. speech-to-speech, streaming vs. batch generation, model size) gets weighed against this budget.
• Turn-taking. In text chat, "your turn" is unambiguous: you send a message, wait for a reply. In voice, the system has to detect when a person has actually finished speaking, not just paused for breath, using voice activity detection and often a semantic signal (does this sound like a complete thought), or it either interrupts too early or leaves an awkward silence.
• Interruption (barge-in) handling. A natural conversation includes people talking over each other. A voice AI that can't be interrupted mid-response, and instead finishes a long answer while the user is trying to redirect it, feels robotic. Handling barge-in means the system must detect new speech input while it is still generating and playing its own output, then gracefully stop and reprocess.
• Streaming generation and playback. Waiting for a full response to generate before starting audio playback adds unacceptable latency; production systems stream generated audio in chunks as it's produced, so playback starts within a few hundred milliseconds even if the full response takes longer.

Real Tools

Most production voice products today are still cascades built from best-in-class pieces (Deepgram for transcription, an LLM for reasoning, ElevenLabs for synthesis) because it offers more control over each stage; native speech-to-speech APIs are gaining share as they mature.

Real-World Showcase

• OpenAI's Realtime API powers voice features in ChatGPT's Advanced Voice Mode, demonstrating sub-second, interruptible, natural-sounding conversation at consumer scale.
• Call-center automation vendors commonly build on Deepgram (transcription) plus ElevenLabs (synthesis) cascades specifically because it lets them swap the underlying LLM independently of the audio pipeline.
• Google's Gemini Live API brings native audio-in, audio-out conversational AI to Android and web products, competing directly with OpenAI's Realtime API on the speech-to-speech architecture.

Try It Yourself

1. Sketch a latency budget for a voice assistant: allocate your target 500ms round trip across audio capture, network transit, model inference, and audio synthesis, and identify which stage is hardest to hit.
2. Compare cascade and speech-to-speech architectures for a customer-service phone bot: which would you pick if you needed to swap in a specialized fraud-detection model mid-call, and why does that favor one architecture over the other?
3. Design a barge-in policy in plain English: what triggers the system to stop talking and start listening, and how much of its own in-progress response should it discard versus try to resume?

Common Mistakes

• Building a voice product on a text-chat latency budget, discovering only in user testing that multi-second pauses feel broken in a way they never did in a chat UI.
• Ignoring turn-taking and shipping a system that either cuts users off mid-sentence or leaves long silences waiting for a "complete" utterance that voice activity detection never confidently signals.
• Choosing a cascade architecture purely for ease of integration without budgeting for the latency it accumulates across three separate model calls.
• Treating interruption handling as a nice-to-have instead of a core requirement, leading to a voice agent that talks over users or ignores attempts to redirect it.
• Testing voice quality only with clean, single-speaker audio and never validating performance with background noise, accents, or overlapping speech, all common in real deployments.

Key Takeaways

• Voice AI is multimodal AI (AI-052) constrained by a hard real-time deadline, which introduces problems text-based multimodal systems never face.
• Speech-to-speech models process audio directly and are faster and more expressive than the older cascade of speech-to-text, LLM, and text-to-speech, but the cascade remains common for its flexibility.
• Latency budget, turn-taking, and interruption handling are first-class architectural concerns, not afterthoughts layered onto a text-based design.
• OpenAI's Realtime API and Google's Gemini Live represent native speech-to-speech; Deepgram and ElevenLabs are best-in-class cascade components.
• Most production systems today are still cascades, chosen for per-stage control, even as native speech-to-speech options mature.