Voice Pipeline Architecture

A voice AI pipeline converts spoken input into a text response and speaks it back — seamlessly enough that the interaction feels conversational. Building one requires careful latency management at every stage, robust interruption handling, and deliberate choices about which components run locally vs in the cloud.

Full Pipeline Diagram

The classic cascade pipeline processes audio sequentially through four stages:

Latency Budget

The target for conversational voice is 800ms time-to-first-word — the delay from when the user stops speaking to when they hear the first word of the response. Beyond ~1,200ms, the interaction starts to feel like a phone system, not a conversation.

Stage 1: Voice Activity Detection (VAD)

VAD detects when the user starts and stops speaking. Without it, you have no clean utterance boundary to feed to STT.

Key VAD tuning parameters: end-of-speech padding (how long to wait after the last speech frame before declaring the utterance complete — 300–600ms is typical; too short causes cutoffs, too long adds latency) and detection threshold(higher = fewer false triggers from background noise).

Stage 2: Speech-to-Text (STT)

For cascade pipelines, Whisper Turbo INT8 on GPU is the recommended local option (fast, accurate, free). For cloud-only or thin clients, Deepgram Nova-2offers excellent accuracy with streaming support and speaker diarisation.

Stage 3: LLM

The LLM receives the STT transcript and conversation history. Key design decisions:

• System prompt for voice persona — keep it short (voice personas do not need the same long instructions as text chatbots); specify speaking style (concise, conversational, no bullet points or markdown)
• Model tier selection — for low-latency voice, a faster/smaller model is often better than a more capable one: GPT-4o mini, Claude Haiku, or Gemini Flash at 400–600ms TTFT vs 800ms+ for top-tier models
• Keep context short — TTFT scales with prompt length; summarise or prune conversation history aggressively for voice (last 3–5 turns)
• Local option — Llama 3.1 8B via Ollama on a GPU machine gives 300–500ms TTFT and costs nothing per token at scale

Stage 4: Text-to-Speech (TTS)

Piper on CPU is intentionally used even on GPU machines to keep the GPU free for the LLM and STT — it is fast enough for voice at 50–100ms and runs without competing for GPU memory. Pre-warm TTS models at application startup to eliminate the 200–800ms cold-start cost on the first request.

Interruption Handling (Barge-In)

A conversational voice agent must handle the user speaking while the agent is talking — called barge-in or interruption. Without it, the user must wait for the agent to finish, which feels unnatural.

Key implementation note: acoustic echo cancellation (AEC) is required on devices without headphones. Without AEC, the microphone picks up the speaker audio, which triggers false VAD detections and creates feedback loops. Most audio SDKs (WebRTC, pyaudio with AEC filter, Apple AVFoundation) include AEC.

On-Device vs Cloud Architecture

Wake Word Integration

Always-on voice assistants need a wake word engine to avoid continuously processing audio:

• openWakeWord — Apache 2.0, custom wake words from audio clips, runs 15–20 models simultaneously on a Raspberry Pi 3 single core; native Home Assistant integration; recommended for privacy-sensitive deployments
• Porcupine (Picovoice) — commercial, >97% accuracy, <1 false alarm per 10 hours, trains new wake words in seconds via web UI; runs on ARM Cortex-M4 microcontrollers; best for commercial products

Wake word engines run continuously on CPU at very low power. Only when the wake word is detected does the full STT → LLM → TTS pipeline activate. The wake phrase itself is stripped from the STT transcript (the user says “Hey Assistant, book a meeting”; the LLM sees “book a meeting”).

Pipeline Failure Modes

Checklist: Do You Understand This?

• What is the time-to-first-word (TTFW) target for a conversational voice agent, and why?
• How does sentence-boundary streaming reduce perceived latency without changing actual computation time?
• What does VAD do, and what happens if end-of-speech padding is set too short?
• Why is Piper TTS run on CPU even when a GPU is available?
• What are the five steps in a barge-in (interruption) handling sequence?
• What is acoustic echo cancellation (AEC) and when is it required?
• For a privacy-sensitive voice application that must work offline, what local stack would you choose?