Why Voice Agents Feel Broken at 800ms

Voice agents are judged on a timescale the rest of the stack never has to think about. A web request that takes 800 milliseconds is fine. A voice agent that takes 800 milliseconds to start replying feels broken, and users respond to it the way they respond to a person who has stopped listening: they repeat themselves.

The useful number is not total response time. It is time to first audio, measured from the moment the user stops speaking to the moment they hear the first sound coming back.

Why first audio is the only number that matters

Once the reply has started, the remaining latency disappears. The rest of the response is generated and spoken while the user is listening, and listening occupies them. Nothing about a long answer feels slow, provided it began promptly.

This inverts the usual optimisation instinct. A system that starts speaking in 400ms and then talks for eight seconds is experienced as fast. A system that thinks for 1.2 seconds and then delivers a tight three-second answer is experienced as slow. Total processing time is worse in the first case and nobody notices.

Human conversation sets the reference. Gaps between turns cluster around 200 milliseconds across languages, which is faster than speech production alone should permit, because listeners predict the end of a turn and prepare a reply before it arrives. Anything under roughly 300 to 500 milliseconds falls inside that expectation. Past about 800, the silence is long enough to be read as a failure to hear, and the user talks over the reply that was about to arrive.

Where the milliseconds actually go

The distribution surprises people. The language model is usually not the largest share.

Stage	Typical contribution	Notes
Endpointing	200–700ms	Often the single largest item, and pure additive wait
Final speech recognition	50–150ms	Most of the transcript already exists if streaming
Model time to first token	150–400ms	Total generation time is largely irrelevant here
Speech synthesis to first audio	80–300ms	Varies enormously by engine and voice
Network round trips	30–150ms	Multiplied by the number of hops between services

These are representative ranges rather than measurements of any one system, and the point is the shape rather than the specific figures: four of the five rows are not the model, and the largest row is a timer.

Endpointing is the hard constraint

Endpointing is the decision that the user has finished speaking. Most implementations wait for a fixed silence threshold, and that wait is added to every turn, unconditionally.

The trade is direct and unpleasant. A short threshold makes the agent cut people off mid-sentence, particularly anyone who pauses to think, which includes most people describing a problem. A long threshold makes every exchange feel sluggish. Both failure modes are described by users as the agent being bad at listening, and tuning the number simply moves the complaint.

This is the part of the budget where engineering effort actually pays. Semantic endpointing, deciding whether an utterance sounds finished rather than waiting a fixed period, is the only approach we know of that improves both sides at once, and it is meaningfully harder than a timer.

Barge-in decides whether it feels like a conversation

Interruption handling matters more to perceived quality than any latency number, and it is routinely treated as a later feature.

If a user starts talking while the agent is speaking, three things must happen fast: audio output stops, the pending response is abandoned, and the incoming speech is captured from its first syllable rather than from whenever the system noticed. Systems that handle the first two and miss the third lose the beginning of every interruption, so the agent asks people to repeat themselves at exactly the moment they were already frustrated.

An agent that responds in 400ms but cannot be interrupted feels worse than one at 700ms that stops cleanly the instant you speak.

What streaming buys, and what it costs

Streaming does not make any stage faster. It overlaps them, so recognition runs while the user is still talking and synthesis starts on the first sentence rather than the last. That overlap is where most of the achievable improvement lives.

The cost is a class of bug that does not exist in batch systems. Recognition revises its transcript as more audio arrives, so a reply can begin from a version of the sentence that the recogniser subsequently changes. The agent has already said something premised on a misheard word. There is no clean fix, only a choice about how long to wait before committing, which is the endpointing trade wearing a different hat.

What does not work

Switching to a faster model first. It is the most visible knob and frequently the wrong one. If endpointing costs 500ms and synthesis takes 250ms to start, halving model latency changes the felt experience very little. Measure the stages before touching the model, the same reasoning that makes model choice a smaller lever than it looks in most production systems.

Filler audio to cover the gap. Playing "let me check that" while the real response generates buys perhaps one turn of goodwill and then becomes irritating, because it is obviously the same clip. It converts a latency problem into a personality problem.

Colocating everything. Worth doing, and the ceiling is low. Network hops are the smallest row in the table for most deployments, so removing them optimises the part that was not hurting.

What we do not know

We do not have a good model of how latency tolerance varies with context. The 300 to 500ms figure comes from conversational norms, but tolerance plainly shifts with what the user believes the agent is doing, a pause before a complex lookup is forgiven in a way an identical pause before a simple answer is not. Whether that is worth engineering around, by signalling effort rather than reducing it, we do not know.

We are also unsure how much of the remaining gap is fixable at all. Some of the budget is irreducible: audio has to be captured, a decision has to be made about whether the turn ended, and sound has to be produced. Our suspicion is that the industry is closer to that floor than the current pace of announcements suggests, and we would rather say that than imply the next model release will close it.

The practical position

Measure time to first audio, broken down by stage, before changing anything. Most teams discover the model is third or fourth on the list.

Spend the effort on endpointing and barge-in, because that is where both the largest number and the sharpest perceived-quality effects are. Accept that some of the budget cannot be recovered, and that a voice agent which stops the instant you speak will forgive a great deal of latency that one which talks over you will not. As with instrumenting an agent you cannot otherwise see, the first move is measuring the thing properly rather than optimising the part that is easiest to change.