The 200ms voice latency budget — where every millisecond goes

The 200ms target

The first time a voice agent responds in under 200 milliseconds end-to-end, it feels like a trick. You say something, and something says something back before you've finished registering that you stopped talking. That sensation — the one that makes testers laugh and say "wait, seriously?" — is the product. Everything else is logistics.

We've been chasing that sensation across dozens of deployments. Twilio, LiveKit, Vapi, custom WebRTC stacks. The path to 200ms is the same every time: enumerate every millisecond, assign a budget, and fight for it in each layer separately. What follows is that accounting.

The turn budget we target is 180–220ms from the moment the end-of-utterance detector fires to the moment the caller hears the first TTS byte. Callers perceive gaps under ~250ms as natural conversational rhythm. Above 400ms you get the awkward pause. Above 700ms they start re-prompting or hanging up.

Speech to text

Budget: 80–110ms. This is almost always your largest line item.

The three levers are provider, model size, and streaming mode. Most teams start with Whisper large-v3 because the accuracy is undeniable. But large-v3 adds 60–80ms over the medium model for a WER improvement you often can't measure in a real telephony context where noise floors and codec artifacts dominate. We default to whisper-1 via OpenAI streaming API, or Deepgram Nova-2 when latency is the primary constraint — Nova-2 consistently clocks under 60ms for sub-five-second utterances.

The bigger win is streaming. Don't wait for a complete audio chunk. Stream raw PCM frames as they arrive and let the STT provider emit partial transcripts. By the time end-of-utterance fires, you often already have 70–80% of the transcript, which lets the LLM pipeline start before transcription is complete.

End-of-utterance detection is its own sub-problem. VAD (voice activity detection) libraries like Silero add 8–12ms and reduce false triggers on breath sounds and filler words. Worth every millisecond — a false trigger wastes the full 200ms budget and irritates the caller.

Network and routing

Budget: 12–24ms. Often invisible until it isn't.

The failure mode is routing a voice call through a general-purpose API gateway that adds 15–30ms per hop. Voice infrastructure needs a dedicated low-latency path. Co-locate your inference servers in the same cloud region as your telephony provider. If you're on Twilio, that means us-east-1 for US calls. GCP us-central1 for midwest-heavy traffic. Measure with ping and a synthetic load test before you commit.

WebSocket connections to your inference layer should be persistent and pre-warmed. Establishing a new TLS connection per turn adds 20–40ms. Keep one connection open per active call session and reuse it for all turns.

LLM inference

Budget: 60–90ms to first token.

Time-to-first-token (TTFT) is what matters, not total completion time. You start the TTS pipeline the moment the first token arrives, so you're racing against TTFT not total latency.

GPT-4o mini and Claude Haiku 3.5 both consistently hit 40–70ms TTFT under moderate load. GPT-4o full runs 90–130ms. For voice, you almost never need the larger model — the turn is short, the context is bounded, and the task is conversational not analytical. Reserve the big model for complex tool calls.

Prompt construction is where teams give away latency for free. Every token in the system prompt costs inference time. We cap voice system prompts at 400 tokens. Dynamic context injection (calendar state, caller history, account data) is retrieved in parallel with STT so it's ready by the time the transcript lands.

Keep max_tokens tight. A voice turn response is 20–60 tokens. Setting max_tokens: 150 and stopping at the first sentence-ending punctuation via a stop sequence saves meaningful generation time on verbose models.

TTS first byte

Budget: 30–50ms.

ElevenLabs Flash, Cartesia Sonic, and OpenAI TTS streaming all hit 25–45ms first-byte latency when co-located. The trap is requesting a full sentence before starting playback. Stream the TTS response and begin audio playback on the first 200ms audio chunk. The caller hears the start of the response before the LLM has finished generating it.

Voice selection matters for perceived latency too. A voice with a slow speaking cadence feels laggy even at 180ms end-to-end. A naturally fast-paced voice at 230ms feels snappier. We run A/B tests on voice cadence as part of every initial deployment.

The four cheats that buy 80ms

These aren't hacks — they're the difference between a 270ms system that feels slow and a 270ms system that feels instant.

Speculation. If your agent follows a predictable script (e.g., appointment booking), you can predict the likely next user turn and pre-generate the probable response before they finish speaking. Hit rate on the first 2–3 turns of a structured flow is typically 70–80%. When you're right, the effective latency is near zero.

Backchannels. Filler phrases ("Got it", "Sure", "One moment") buy 400–800ms while real generation runs. Use them when the tool call or retrieval will take more than 300ms. The trick is making them sound natural and turn-specific, not generic. We generate them dynamically at the same time as the tool call fires.

Pre-warming LLM context. Keep a rolling context window pre-loaded in GPU memory for active calls. Re-initialising KV cache per turn adds 15–25ms at the inference layer.

Parallel execution. STT → tool retrieval → LLM → TTS is not a linear pipeline. STT streaming overlaps with tool retrieval. TTS streaming overlaps with LLM generation. Diagram your call graph and identify every place two operations can start before the first completes.

What feels human means

The 200ms target is not arbitrary. It's the upper bound of what conversational psychologists call the "floor transfer offset" — the gap between turns in natural human conversation, which averages 200ms globally across languages and cultures.

Below 200ms: feels instantaneous. 200–400ms: natural. 400–700ms: slow but acceptable. Above 700ms: the caller starts doubting whether the system heard them.

But raw latency is only part of the equation. Prosody — the rhythm, pitch, and cadence of the synthesized voice — accounts for roughly half of the perceived naturalness score in every listening test we've run. A 180ms response in a robotic voice feels worse than a 280ms response in a warm, natural voice.

The most human-feeling systems we've shipped are not the fastest. They're the ones where every layer — from VAD sensitivity to TTS voice selection to backchannel timing — was tuned together rather than optimised in isolation.