Mixture of Experts: Cheaper per Token, Not per GPU

Mixture-of-experts models are often summarised as "bigger but cheaper". That is half right, and the half that is wrong is the half that decides your hardware bill.

A sparse mixture-of-experts model is cheaper per token in computation. It is not cheaper per GPU in memory. Whether it is a good choice depends almost entirely on which of those two you are short of.

How it works

In a dense model, every parameter takes part in processing every token. In a mixture-of-experts model, some layers are split into several parallel sub-networks, the experts, and a small router decides which few of them handle each token.

The idea is old. A 2017 paper on sparsely-gated mixture-of-experts layers reported increasing model capacity by more than a thousandfold with only minor losses in computational efficiency, by running only part of the network for each example. The Switch Transformer work later describes the result as a model with an outrageous number of parameters but a constant computational cost.

How the router decides

The router is a small learned layer. For each token, at each mixture-of-experts layer, it scores every expert and sends the token to the highest-scoring few. Mixtral sends each token to two experts and combines their outputs. The Switch Transformer simplified this further by routing each token to a single expert.

Two details matter in practice. First, the choice is made per token and per layer, so a single sentence can pass through many different experts on its way through the network; the Mixtral paper notes that the selected experts can be different at each timestep. Second, a router left to itself tends to favour a few experts and starve the rest. Training therefore adds pressure to balance the load across experts, because an expert that never receives tokens is memory spent on nothing.

What an "expert" is, and is not

The name invites a misreading: that one expert handles code, another French, another mathematics. That is not how these models are built, and it is not what has been observed.

An expert is simply one of several feedforward blocks in a layer, and the router learns which to use without being told about topics. When the Mixtral authors analysed which experts were chosen for text from different domains, they report finding no obvious assignment of experts by topic. Whatever the experts specialise in, it is not something that maps neatly onto human subject areas.

This matters because it rules out a tempting optimisation, loading only the experts "relevant" to your domain. You cannot know in advance which experts a given input will use, so all of them stay resident.

The numbers that matter

Mixtral 8x7B is a convenient public example. Each of its layers has eight experts, and the router picks two of them for every token. The paper puts the result plainly: each token has access to 47 billion parameters but uses only about 13 billion active parameters during inference.

The following is illustrative arithmetic on those figures, at 16-bit precision, ignoring activations and the KV cache:

	Memory to hold the weights	Compute per token
Dense 13B model	≈ 26 GB	13B parameters' worth
Mixtral 8x7B	≈ 94 GB	≈ 13B parameters' worth
Dense 47B model	≈ 94 GB	47B parameters' worth

The mixture-of-experts model computes like the small dense model and occupies memory like the large one. That is the whole trade in one table.

Where the trade pays

High-volume serving on hardware you already have. If you are provisioned for a large model's memory anyway, a mixture-of-experts model gives you the capability of more parameters while spending the compute of fewer. The Mixtral authors report it matching or outperforming Llama 2 70B and GPT-3.5 on the benchmarks they evaluated, at a fraction of the active parameters.

Throughput-bound workloads. When the GPU's arithmetic is the bottleneck, doing less arithmetic per token is a direct saving. This is the regime large providers operate in, and part of why the architecture spread so quickly.

Where it does not

When memory is the constraint. On a single GPU, a laptop, or an edge device, the binding limit is usually how much fits, not how fast it computes. There the relevant comparison is with a dense model of the same memory footprint, and that model is simpler to serve and fine-tune.

When batches scatter. Different tokens go to different experts, so at scale the experts are often spread across devices and tokens have to be shuffled between them. The Switch Transformer authors name complexity, communication cost and training instability as the reasons mixture-of-experts adoption was held back. Those costs have not disappeared; they have been engineered around by teams with the scale to do it.

When you have not measured. Benchmark parity with a larger dense model does not imply parity on your task, any more than a leaderboard predicts which model wins in production.

Quantisation does not change the shape of the trade

Storing weights at lower precision shrinks memory for any model, and mixture-of-experts models are no exception. As illustrative arithmetic on Mixtral's 47 billion parameters, ignoring overheads:

Precision	Approximate weight memory
16-bit	≈ 94 GB
8-bit	≈ 47 GB
4-bit	≈ 24 GB

Every row still holds all eight experts. Quantisation moves the whole table down; it does not close the gap between what must be stored and what is computed, which remains roughly three and a half to one. Lower precision also has a quality cost that varies by model and task, and should be measured rather than assumed.

A decision table

Your situation	Usually the better fit
Using a hosted API	Whatever the provider runs; judge on price, latency and output
One GPU or a local machine, memory is tight	A dense model sized to the memory
Narrow, repeatable task	A small dense model, often fine-tuned
High-volume serving, memory already provisioned, compute is the bottleneck	A mixture of experts, tested against a dense alternative
You need to fine-tune often	A dense model is simpler to adapt and serve

What to measure before switching

If you are considering replacing a dense model with a mixture of experts on your own hardware, four measurements settle it better than any parameter count:

1. Tokens per second at your real batch size. Measure at the concurrency you actually serve, not at one request, since routing behaviour changes with batch size.
2. Memory headroom after the weights. Whatever is left has to hold the KV cache for every concurrent request, and a model that fits with no room for the cache does not really fit.
3. Quality on your own evaluation set. Benchmark parity is a starting hypothesis, not a result.
4. Cost per million generated tokens on your hardware. This combines the first three into the number the decision actually turns on.

If the mixture of experts wins on the fourth without losing on the third, switch. If it only wins on paper, do not.

You may not need to decide this

If you use a model through an API, the provider has made this choice and priced it in. The architecture matters only indirectly, through price and latency.

If you host models yourself, start from your constraint rather than the architecture. Short of memory, a smaller dense model is usually the better fit, and for narrow tasks a small specialised model often beats both. Short of compute with memory to spare, a mixture of experts is worth testing. Remember too that weights are only part of the memory bill: the KV cache grows with every request, whichever architecture you pick.

What we do not know

The active-parameter count describes arithmetic, not wall-clock speed. Real throughput depends on how tokens spread across experts, how experts are placed on devices, and how well the serving stack batches them, and those effects vary enough between deployments that we would not predict a speed-up from the parameter counts alone. Measure it on your hardware, at your batch sizes.

The figure in this note may be reused with attribution and a link to this page.