Small Models Are Winning the Fights That Matter

Every model release is measured against the same leaderboard: broad reasoning, broad knowledge, broad everything. It's a reasonable way to compare frontier models to each other. It's a bad way to decide what to run in production, because almost nothing in production needs broad.

Most production tasks are narrow

Classify this support ticket into one of twelve categories. Extract the invoice number and date from this document. Decide which of three tools to call based on this request. Route this message to the right team.

None of that requires general reasoning across unfamiliar domains. It requires consistent, correct behavior on a task whose shape you already know, because you defined it. That's a fundamentally different problem than "answer any question a user might ask," which is what frontier models are built and priced for.

A model with a few billion parameters, fine-tuned or carefully prompted for exactly that narrow task, routinely matches a much larger model's accuracy on it. Not because it's smarter in general. Because the task doesn't require general intelligence, and the large model's extra capacity goes unused on it.

The cost difference isn't marginal

Inference cost scales with model size: more parameters means more compute per token, which means more cost per call. The gap between a small specialized model and a frontier model on a per-call basis is commonly an order of magnitude, sometimes more.

At low volume, that difference doesn't matter enough to think about. At production volume, a few million calls a month, it's the difference between a workload that's economically sustainable and one that isn't. Teams that route every request, regardless of complexity, to the largest available model are frequently paying frontier prices for tasks a much cheaper model already handles correctly.

Latency is a hard constraint, not a preference

Cost is a spreadsheet problem. Latency is a product problem: users abandon interfaces that feel slow, and "feel slow" has a fairly consistent threshold across interactive products, somewhere in the low hundreds of milliseconds for something that's supposed to feel responsive.

Large models are slower per token, simply because there's more computation to do per step. If a product has a tight latency budget, a large model can be the wrong choice even in cases where it would give a marginally more polished answer, because the user experience cost of the delay outweighs the quality gain. This shows up most obviously in voice interfaces and autocomplete-style features, where the acceptable delay is measured in tens of milliseconds, not seconds.

Where large models still win, clearly

None of this is an argument that small models are simply better. On tasks that require synthesizing information across unfamiliar domains, holding a long and complex context, or reasoning through a problem with no fixed shape, a frontier model's extra capacity is doing real work, and a small model's accuracy gap on that kind of task is not marginal. Open-ended research assistance, complex multi-step reasoning, and novel problem-solving are where the size actually earns its cost.

The mistake isn't using large models. It's using them by default, for everything, without checking whether the task in front of you is one of the narrow ones that never needed that capacity.

A decision rule, not a rule of thumb

The practical question for any given task: can you write down, in advance, what a correct answer looks like well enough to evaluate it? If yes, the task is narrow enough that a small model is worth testing against, because narrow, evaluable tasks are exactly where small models close the gap.

If the answer is "it depends on context I can't fully specify," the task is closer to open-ended, and that's where the extra capacity of a larger model is more likely to be earning its cost rather than sitting idle.

Routing beats picking one model for everything

The systems that get this right in production rarely choose one model for the whole workload. They route: a small model handles the high-volume, well-defined majority of requests, and only the genuinely ambiguous or complex cases escalate to something larger. This isn't a compromise position. It's the version of the architecture that spends compute where the task actually needs it, instead of spending frontier-model money on requests a much smaller model was already getting right.

The router itself has to be cheap and fast, since it sits in front of every request and adds its own latency before either model even starts. A router that's slower or more expensive than the savings it produces defeats the point, which is why the classification step deciding "small or large" is usually the smallest, most narrowly scoped model in the whole system, not a separate large model making the routing decision.

The harder part in practice isn't building the router. It's defining the boundary correctly in the first place, deciding, for a given product, which requests are genuinely narrow and which only look narrow until a user asks something the small model wasn't built for. Get that boundary wrong in one direction and you're overpaying for capacity you don't use. Get it wrong in the other direction and the small model quietly returns a confident wrong answer on a request it was never suited to handle, which is a worse failure than routing it to the expensive model would have been.

The leaderboard measures which model is smartest in general. Production doesn't ask that question. It asks which model is right for this specific call, and increasingly, for most calls, the answer isn't the biggest one available.

This is also a cost story that starts before inference. Training the frontier model in the first place buys less than the announcement implies: the same discipline that says "match the model to the task" at inference time applies to whether you needed frontier scale at all. Model selection and infrastructure decisions are the same question asked at different points in the pipeline, which is part of what this lab's work on agents in production keeps running into.