Which Model Should You Actually Run in Production

The question is usually asked as which model is best, and at that level of generality it has no answer. The models at the top of a general leaderboard are not reliably the models that win on any particular narrow task, because the leaderboard is measuring average performance across a distribution of problems your production traffic does not resemble.

The answerable question is narrower: which model performs best on your workload, inside your latency budget, at your cost per call. That comparison has to be run, and running it takes less time than most teams spend deliberating instead.

Leaderboards measure the wrong distribution

A general benchmark samples broadly, across reasoning, knowledge, code, mathematics, and reports a number that summarises performance over all of it. Breadth is the point.

Production is the opposite. A support classifier sees support tickets. An extraction pipeline sees invoices. A routing layer sees the same twelve intents in slightly different words, forever. Performance on a broad distribution predicts performance on a narrow one only loosely, and the looseness runs both directions: a model can lead a leaderboard and lose on your task, or trail it and win.

This is why benchmarks are useful for exclusion and misleading for selection. They will tell you a model is not serious. They will not tell you which of three serious candidates to run.

The four axes that actually decide it

Axis	What to measure	Why it decides
Task accuracy	Performance on your examples, not a benchmark	The only capability number that transfers
Latency	Time to first token, and the tail under load	Averages hide the p99 that users actually feel
Cost	Per call at your real token volume	Per-token pricing misleads once context grows
Controllability	Structured output adherence, refusal behaviour	Determines how much prompt scaffolding you maintain

The last row is the one most often left out and the one that most often decides the outcome. Two models with indistinguishable accuracy can differ sharply in how reliably they return the shape you asked for, and a model that honours a schema without coaxing removes an entire class of parsing and repair code. That is a bigger engineering difference than a couple of points on a benchmark.

Latency deserves the same scrutiny. The average is close to meaningless; what users experience is the tail, and tails diverge under load in ways a quiet afternoon of testing will not reveal. If the output is spoken rather than read, time to first token dominates everything else in the budget.

Build the evaluation before you pick

The comparison is cheap and almost nobody does it.

Take fifty to a hundred real examples from actual traffic. Include the awkward ones, the ambiguous phrasing, the truncated input, the case nobody could agree on, because those are where models diverge and where a curated set silently lies to you. Write down what a correct response looks like for each.

Then run every candidate against that set, unchanged, and compare. That is the whole method.

The set outlasts every model in it. Frontier models are replaced constantly; a hundred labelled examples from your own domain keep their value for years and turn each new release from a judgement call into a measurement. Teams that have one upgrade in an afternoon. Teams that do not argue about it for weeks and then switch on vibes.

What does not work

Vibe-checking in a chat window. Twenty prompts typed by the person who wants a particular answer is not evidence. It oversamples the easy middle of the distribution and misses precisely the inputs that break things.

Trusting vendor comparisons. Every published comparison is run by someone with a preferred outcome, using prompts tuned for their model. This is not usually dishonest, it is just that prompt effort is not distributed evenly, and the effect is large enough to reverse a ranking.

Upgrading the model to fix a broken feature. The common failure is retrieval returning the wrong context, a prompt that underspecifies the output, or a tool contract that drifted. A stronger model given wrong context produces a more confident wrong answer, and none of that surfaces as an error. Establish where the failure originates first.

Optimising for the largest model you can afford. Cost and latency are constraints, not preferences, and for narrow repeatable work a smaller model frequently matches a frontier one at a fraction of both.

What changes, and what does not

Specific model recommendations go stale within months. Anything written today naming a particular winner will be wrong by the time it is read, which is why this post does not name one.

What does not go stale is the method and the axes. Task accuracy on your own examples, latency at the tail, cost at real volume, and how much scaffolding controllability demands, those four have decided this question for as long as there have been options, and they will decide the next one.

What we do not know

We do not have a good way to predict, before running the evaluation, which tasks will show large differences between models and which will show none. Some narrow tasks separate candidates sharply and others produce a dead heat, and we cannot currently tell which is which except by measuring.

We are also uncertain how much of the observed difference between frontier models is genuine capability rather than prompt-format sensitivity. A model that appears worse sometimes only wants the instruction phrased differently, and distinguishing the two properly takes more effort than most comparisons, including most of ours, actually spend.

The practical position

Stop asking which model is best and start asking which one wins on your hundred examples. Build the set once, from real traffic, including the ugly cases.

Then treat model selection as a measurement you re-run rather than a decision you agonise over — the same way a narrow production task often turns out not to need a frontier model at all. The set is the durable asset. The model is a swappable part.