Token choice vs expert choice routing in MoE
In a mixture-of-experts layer, something has to decide which tokens each expert processes. Token choice vs expert choice routing is the question of who makes that call.
It sounds like an implementation detail. It is not: token choice vs expert choice routing decides whether the model fights load imbalance, dropped tokens, or broken causality, because each scheme guarantees exactly one thing.
Token choice lets every token pick its experts. Expert choice inverts it: every expert picks its tokens. Everything else about the comparison follows from that inversion.
Who picks whom
In token choice, the router scores all experts for each token and sends the token to its top k. Switch Transformer routes top-1, GShard top-2, and OLMoE top-8 out of 64 experts.
Every token gets exactly k experts. How many tokens each expert gets is whatever the router's preferences add up to, and that is where the trouble starts.
Fanout's mixture-of-experts routing explainer covers the gating math behind those scores.
Expert choice routing flips the direction. Each expert selects the top k tokens that prefer it, with k = nc/e: n tokens in the batch, c the capacity factor, e the number of experts.
Work the formula once. A 4,096-token batch, 64 experts, and a capacity factor of 2 gives each expert exactly 128 tokens per layer, no matter what the router scores say.
Compute per token becomes variable instead. In the paper's runs, most tokens landed on one or two experts, 23 percent on three or four, and about 3 percent on more than four.
What each direction guarantees
Expert choice guarantees load balance by construction. Every expert fills the same fixed bucket every step, so no bucket overflows and none goes empty.
The guarantee matters because experts are sharded across GPUs, and the busiest shard sets the step time. Fanout's expert parallelism guide works through that failure mode.
Token choice guarantees nothing about balance. OLMoE measured what happens without correction: early in training, the router sent nearly all tokens to a single expert.
So token-choice models buy balance with auxiliary objectives. OLMoE trains with a load-balancing loss at coefficient 0.01 plus a router z-loss at 0.001. The z-loss costs about 2 percent throughput and buys fewer loss spikes.
The auxiliary loss is a soft constraint, which cuts both ways. It nudges the router toward balance without forbidding specialization outright, but it never guarantees the balance expert choice gets for free.
Balance has a number attached. With 8 experts chosen per token out of 64, a balanced router assigns each expert one eighth of the tokens in the batch, and the load-balancing loss penalizes the gap between that target and reality.
Dropped tokens flow in opposite directions
Give token choice a fixed per-expert capacity and popular experts overflow. Tokens that arrive after the buffer fills skip the expert entirely and pass through the layer on the residual stream.
The expert choice paper measured this on GShard-style top-2 routing: some experts run 20 to 40 percent over capacity, which means real tokens losing their expert compute at scale.
Dropless token choice removes the fixed buckets and computes exact assignments instead, which is the route OLMoE takes. The imbalance cost moves into kernel and communication efficiency rather than lost tokens.
Expert choice never overflows an expert. Its failure mode is the mirror image: a token can be picked by many experts, or by none. OLMoE notes that tokens no expert selects simply lose compute, and performance suffers for them.
The autoregressive catch
Expert choice has a structural problem with text generation. The top-k selection runs across the tokens in the batch, so an expert choosing its tokens compares past and future positions against each other.
The authors say so themselves: the method "might not immediately apply to auto-regressive text generation" because the implementation "takes in the past and future tokens" to perform the selection.
At decode time the future does not exist. Routing computed one token at a time no longer matches the routing the model saw in training, and that train-inference mismatch degrades generation.
Batching adds a second mismatch. Which experts pick a token depends on what else shares the batch, so the same prompt can route differently under different traffic.
This is the main reason open decoder MoEs, OLMoE and Kimi K3 among them, route by token.
Expert choice stays a strong fit for encoders and for models that see the full sequence on every pass.
EC-DiT scales diffusion transformers with adaptive expert choice for exactly that reason. Denoising processes whole sequences at once, so there is no future position to leak.
What the measurements say
Both schemes have wins on paper. The expert choice paper reports reaching GShard top-2's perplexity in less than half the training steps, with each step about 20 percent faster.
OLMoE ran the comparison inside a dropless token-choice setup and got the opposite ordering: token choice beat expert choice on downstream tasks at the same token budget.
Expert choice still ran about 20 percent faster there, 29,400 versus 24,400 tokens per second per device. The authors took the quality win over the throughput win.
The two results are consistent once you look at the baselines. Expert choice beat a capacity-constrained token choice that was dropping 20 to 40 percent of some experts' tokens.
Give token choice dropless kernels and a balancing loss, and the dropped-token weakness disappears while the consistency advantage remains.
A 2025 survey of MoE in LLMs sorts the trade the same way: expert choice buys controlled balance, token choice buys behavior that matches generation.
Token choice vs expert choice routing, in short
Pick by what the deployment cannot tolerate.
- Decoder-only model that generates text: token choice. Pay for balance with an auxiliary loss, or go dropless.
- Encoder, bidirectional, or diffusion model: expert choice. Balance is free and the full sequence is always available.
- Token choice dropping too many tokens under fixed capacity: try dropless kernels before switching schemes.
- Reading a new MoE paper: check who picks whom first. Most routing claims only hold on one side of that line.