THREAD 02
Language Models
Inference economics, retrieval, model routing, and the boundary between frontier and specialised models.
Open question
Where does additional model capability justify its latency and compute cost?
How we investigate
Worked calculations, literature synthesis, controlled evaluations, and systems profiling.
3 min readWhere the Tokens Actually Go in Long-Context Inference
A transparent look at KV-cache memory, bandwidth pressure, and why a large context window is not the same as reliable recall.
Language ModelsRead
2 min readThe 200ms Turn-Taking Problem in Streaming Voice AI
A practical latency budget for voice agents, and why endpointing—not only model speed—determines whether a conversation feels responsive.
Voice SystemsRead
2 min readWhy RAG Fails When the Chunk Score Is High
Similarity can retrieve a topically related chunk that does not support the answer. Here is how to test retrieval for evidence, not proximity.
RetrievalRead
5 min readWhich Model Should You Actually Run in Production
Leaderboards rank models on a distribution your workload does not resemble. The four axes that decide it, and how to build the comparison that matters.
Language ModelsRead
5 min readWhy Voice Agents Feel Broken at 800ms
Total latency is the wrong number to optimise. What decides whether a voice agent feels alive is time to first audio, and where those milliseconds go.
Voice SystemsRead
4 min readHow AI Overviews Choose What to Cite
Ranking and citation are different selections made by different systems. Why a page can sit in the top ten for a query and never once be quoted.
Generative Engine OptimizationRead
4 min readSmall Models Are Winning the Fights That Matter
Most production tasks are narrow. A decision rule for when a small specialised model beats a frontier one on cost, latency and accuracy.
Language ModelsRead