THREAD 02
Language Models
Inference economics, retrieval, model routing, and the boundary between frontier and specialised models.
Open question
Where does additional model capability justify its latency and compute cost?
How we investigate
Worked calculations, literature synthesis, controlled evaluations, and systems profiling.
7 min readWhat It Costs to Run an LLM, Per Token
Cost per token is GPU price divided by useful tokens per second. Utilisation, batching, prompt length and model choice move it more than the hourly rate.
AI-Native TopologyRead
7 min readWhat Speculative Decoding Actually Saves
Speculative decoding lets a small model draft tokens that a large one checks in a single pass. The output is unchanged, and the speed-up depends on one thing.
Language ModelsRead
7 min readWhat Quantization Actually Costs You
Quantization stores a model's weights in fewer bits, so it fits in less memory and often runs faster. Where the quality goes, and how to measure it.
Language ModelsRead
6 min readYou May Not Need a Vector Database
Most retrieval systems start with a dedicated vector database they do not yet need. What exact search, pgvector and approximate indexes actually trade.
RetrievalRead
6 min readMixture of Experts: Cheaper per Token, Not per GPU
A mixture-of-experts model runs only a few experts per token, so compute falls. Every expert still has to sit in memory. Where that trade pays off.
Language ModelsRead
6 min readWhat the KV Cache Actually Costs You
The KV cache makes generation fast and quietly decides how many requests a GPU can serve. Why it grows with every token, and what shrinks it.
Language ModelsRead
5 min readYour Context Window Is Smaller Than Advertised
A context window is the most a model will accept, not the length it reliably uses. What long-context benchmarks found, and how to size prompts around it.
Language ModelsRead
3 min readWhere the Tokens Actually Go in Long-Context Inference
A transparent look at KV-cache memory, bandwidth pressure, and why a large context window is not the same as reliable recall.
Language ModelsRead
7 min readThe 200ms Turn-Taking Problem in Streaming Voice AI
A latency budget for voice agents, and why deciding when the user has finished, not only model speed, makes a conversation feel responsive.
Voice SystemsRead
3 min readWhy RAG Fails When the Chunk Score Is High
Similarity can retrieve a topically related chunk that does not support the answer. Here is how to test retrieval for evidence, not proximity.
RetrievalRead
5 min readWhich Model Should You Actually Run in Production
Leaderboards rank models on a distribution your workload does not resemble. The four axes that decide it, and how to build the comparison that matters.
Language ModelsRead
5 min readWhy Voice Agents Feel Broken at 800ms
Total latency is the wrong number to optimise. What decides whether a voice agent feels alive is time to first audio, and where those milliseconds go.
Voice SystemsRead
4 min readHow AI Overviews Choose What to Cite
Ranking and citation are different selections made by different systems. Why a page can sit in the top ten for a query and never once be quoted.
Generative Engine OptimizationRead
5 min readSmall Models Are Winning the Fights That Matter
Most production tasks are narrow. A decision rule for when a small specialised model beats a frontier one on cost, latency and accuracy.
Language ModelsRead