The 200ms Turn-Taking Problem in Streaming Voice AI

A voice agent can use a fast model and still feel slow. The missing interval is often endpointing: deciding that the speaker has yielded the floor.

“200ms” is best treated as an engineering target for a responsive handoff. It is not a universal boundary in human perception, nor is it the budget for an entire generated answer.

Decompose the delay

The first audible response arrives after several stages:

endpoint decision + network + transcription + model first token + synthesis startup + playback buffer

Optimising only model inference leaves the other terms intact. Instrument every stage on a shared trace and report percentiles, not only averages. Mobile jitter and noisy rooms make tail latency especially important.

Model choice still matters once the rest of the pipeline is visible. Why Voice Agents Feel Broken at 800ms examines the larger response path, while Which Model Should You Run in Production? explains why a routed, smaller model can outperform a more capable model when time to first token is the binding constraint.

Why silence is ambiguous

A voice activity detector distinguishes speech-like frames from non-speech. It does not know whether a thought is complete. Speakers pause to breathe, search for a word, emphasise a phrase, or wait while reading. A short timeout reacts quickly but interrupts more often; a long timeout protects pauses but creates dead air.

Endpointing should combine signals:

• acoustic activity and trailing-silence duration;
• prosody, including pitch and energy trajectory;
• partial transcript syntax and semantic completeness;
• dialogue state, such as whether the system asked for a short value;
• an explicit interruption path if the prediction is wrong.

Use asymmetric decisions

The agent does not need to commit everything at once. It can acknowledge quickly, begin speculative planning internally, and delay irreversible speech until confidence rises. If synthesis begins speculatively, its audio should remain cancelable before playback.

A robust state machine separates listening, possible_endpoint, committed_turn, speaking, and interrupted. Log the evidence and threshold behind each transition. That makes false cut-offs diagnosable instead of anecdotal.

Evaluation that reflects conversation

Build an audio set containing complete turns, within-turn pauses, fillers, corrections, barge-ins, noise, and different speaking rates. For each case, measure:

1. time from actual turn completion to endpoint commitment;
2. false endpoint rate during a continuing turn;
3. first-audio latency after commitment;
4. successful barge-in and cancellation time;
5. recovery quality after an incorrect decision.

No single silence threshold wins every case. The useful target is a low-delay decision system that knows when it is uncertain and can recover gracefully.

Production guardrails

Keep endpoint confidence separate from transcript confidence. A pipeline may be certain the speaker stopped but uncertain what was said. In that case a clarification is safer than confidently acting on a partial transcript. Likewise, do not let speculative work create external side effects until the turn is committed.

Segment metrics by language, device class, noise condition, and conversation state. An aggregate latency number can hide a system that works in a quiet desktop test and repeatedly cuts off mobile callers. Review recorded failures with consent and retention controls, then promote recurring patterns into the evaluation set.

Finally, make interruption a first-class capability. Stop playback quickly, cancel unnecessary generation, preserve the useful state, and acknowledge the new user turn. Natural timing depends as much on graceful correction as on the first endpoint decision.