The 200ms Turn-Taking Problem in Streaming Voice AI

A voice agent can use a fast model and still feel slow. The missing interval is often endpointing: deciding that the speaker has yielded the floor.

“200ms” is best treated as an engineering target for a responsive handoff. It is not a universal boundary in human perception, nor is it the budget for an entire generated answer.

Decompose the delay

The first audible response arrives after several stages:

endpoint decision + network + transcription + model first token + synthesis startup + playback buffer

Optimising only model inference leaves the other terms intact. Instrument every stage on a shared trace and report percentiles, not only averages. Mobile jitter and noisy rooms make tail latency especially important.

Model choice still matters once the rest of the pipeline is visible. Why Voice Agents Feel Broken at 800ms examines the larger response path, while Which Model Should You Run in Production? explains why a routed, smaller model can outperform a more capable model when time to first token is the binding constraint.

Why silence is ambiguous

A voice activity detector distinguishes speech-like frames from non-speech. It does not know whether a thought is complete. Speakers pause to breathe, search for a word, emphasise a phrase, or wait while reading. A short timeout reacts quickly but interrupts more often; a long timeout protects pauses but creates dead air.

Endpointing should combine signals:

• acoustic activity and trailing-silence duration;
• prosody, including pitch and energy trajectory;
• partial transcript syntax and semantic completeness;
• dialogue state, such as whether the system asked for a short value;
• an explicit interruption path if the prediction is wrong.

Use asymmetric decisions

The agent does not need to commit everything at once. It can acknowledge quickly, begin speculative planning internally, and delay irreversible speech until confidence rises. If synthesis begins speculatively, its audio should remain cancelable before playback.

A robust state machine separates listening, possible_endpoint, committed_turn, speaking, and interrupted. Log the evidence and threshold behind each transition. That makes false cut-offs diagnosable instead of anecdotal.

Evaluation that reflects conversation

Build an audio set containing complete turns, within-turn pauses, fillers, corrections, barge-ins, noise, and different speaking rates. For each case, measure:

1. time from actual turn completion to endpoint commitment;
2. false endpoint rate during a continuing turn;
3. first-audio latency after commitment;
4. successful barge-in and cancellation time;
5. recovery quality after an incorrect decision.

No single silence threshold wins every case. The useful target is a low-delay decision system that knows when it is uncertain and can recover gracefully.

Where the 200ms comes from

People answer each other quickly. The cross-language study by Stivers and colleagues compared turn-taking in ten languages and found that every one of them shows a general avoidance of overlapping talk and a minimisation of silence between turns. The average gap between turns differed across the languages, but only within a band of about 250 ms around the cross-language mean. The finding that matters for voice agents is the shape, not one number: human responses arrive within a few hundred milliseconds, which is quicker than a person could plan a reply from scratch, so listeners must be predicting the end of the turn while the speaker is still talking.

A voice agent that waits for certainty before it starts working cannot match that. By the time it is sure the speaker has finished, a person would already have replied. That gap is what users experience as the agent being slow, even when every component is fast. It is also why "200ms" is a useful shorthand: it names the scale on which a conversation feels natural, so a budget measured in seconds is plainly outside it.

A budget you can write down

Treat the target as a budget and divide it between the stages. The numbers below are illustrative, a way to see which stages dominate, not measurements of any product:

Stage	Illustrative share of the budget	What mainly controls it
Endpoint decision	Large and variable	Silence threshold, model, how much evidence you wait for
Network, uplink	Small to moderate	Distance, mobile jitter, codec
Transcription of the final words	Moderate	Streaming recogniser, how early partial results are used
Model time to first token	Moderate to large	Model size, prompt length, caching, queueing
Speech synthesis startup	Moderate	Streaming synthesis, first-chunk size
Network, downlink and playback buffer	Small to moderate	Jitter buffer size, device audio path

Two things follow. First, no single stage can be optimised alone: a fast model behind a long silence timeout is still slow. Second, the endpoint decision is the only stage where waiting longer is a deliberate choice. Every other stage is trying to be faster. Endpointing is a trade between delay and false interruptions, which makes it the stage most worth designing carefully.

For the network share of the budget, the long-standing telephony reference is ITU-T G.114, which covers one-way transmission time. It is commonly summarised as treating delays up to about 150 ms one-way as acceptable for most conversational use, with quality degrading beyond that. We are relying on those summaries rather than quoting the recommendation's text, so check the current document before using a figure in a specification. The practical lesson is that the network alone can use a large share of any end-to-end target on a mobile connection.

Overlap the stages instead of waiting for each

A sequential pipeline pays every stage's delay in full. A streaming one overlaps them:

• Stream the transcript. Partial recognition results arrive while the user is still speaking, so the model can see most of the turn before it ends.
• Prepare before committing. Retrieval, tool lookups that have no side effects, and prompt assembly can start on a likely endpoint and be thrown away if the user keeps talking.
• Stream the answer. Start synthesis on the first sentence fragment instead of waiting for the full reply.
• Keep buffers small. Playback and jitter buffers are latency you chose; size them for the worst network you must support, not a safe margin on top of that.

Speculative work is only safe when it can be cancelled. Anything with an external side effect, such as sending a message or changing a record, waits until the turn is committed.

How the endpointing approaches compare

Approach	How it decides	Strength	Weakness
Fixed silence timeout	Ends the turn after N ms without speech	Simple, predictable	Cuts off slow speakers, adds dead air for fast ones
Voice activity detector with a hangover	Waits after the last speech frame	Robust to short noise	Still blind to whether the thought is finished
Prosody	Watches pitch and energy falling	Captures how people signal the end of a turn	Varies by language and speaker; needs tuning data
Transcript-based	Judges whether the words so far are a complete utterance	Understands syntax and meaning	Adds model latency; depends on transcript quality
Combined	Merges the signals above with dialogue state	Best accuracy for the delay	Most work to build and evaluate

Open-source voice activity detectors such as the WebRTC detector and Silero VAD solve the first half of the problem: whether there is speech in a frame. They do not decide whether a turn is over. A system built on one still needs a policy on top of it.

The two ways to be wrong

An endpointing system makes two kinds of error, and they cost different amounts.

• Too early cuts the speaker off mid-thought. The user has to repeat themselves, the transcript is incomplete, and trust drops quickly.
• Too late adds dead air after every turn. It is annoying, but nothing is broken and the user can simply wait.

Because of this asymmetry, an agent that is unsure should usually prefer to be slightly late or to respond in a way that is cheap to undo, such as a short acknowledgement, rather than commit to a long answer based on a half-heard sentence. The right operating point depends on the task. A system collecting a phone number can end the turn quickly once it has the right number of digits; one taking a complaint should give the speaker room to pause.

Log every endpoint decision with the evidence behind it. A false cut-off that cannot be reproduced cannot be fixed, and the only way to tune the threshold is against real recorded failures.

Production guardrails

Keep endpoint confidence separate from transcript confidence. A pipeline may be certain the speaker stopped but uncertain what was said. In that case a clarification is safer than confidently acting on a partial transcript. Likewise, do not let speculative work create external side effects until the turn is committed.

Segment metrics by language, device class, noise condition, and conversation state. An aggregate latency number can hide a system that works in a quiet desktop test and repeatedly cuts off mobile callers. Review recorded failures with consent and retention controls, then promote recurring patterns into the evaluation set.

Finally, make interruption a first-class capability. Stop playback quickly, cancel unnecessary generation, preserve the useful state, and acknowledge the new user turn. Natural timing depends as much on graceful correction as on the first endpoint decision.