Mixture of experts routing explained

Mixture of experts routing explained plainly: a small learned projection scores every expert for each token, keeps a few scores, and combines those experts' outputs.

The experts are feed-forward networks inside a transformer block. They are not separate chatbots, and the router does not read a topic label such as "math" or "code."

Routing is simple for one token. It becomes difficult across a batch because expert choices determine compute balance, buffer sizes, and network traffic.

Mixture of experts routing explained from one token

A transformer first produces a hidden-state vector for a token. In a sparse MoE block, a linear router projects that vector to one score per routed expert.

With four experts, the router output has four logits. A common top-k router keeps the largest one or two values and masks the rest.

The sparsely gated MoE paper describes a trainable gate that selects a sparse combination of feed-forward expert networks.

The selected scores become mixing weights through softmax. Each chosen expert processes the token, and the block adds their weighted output vectors.

This replaces the dense feed-forward sublayer. Attention and the residual path still run for every token.

The router repeats this decision at each MoE layer. The same token can select different experts later because its hidden state has changed.

A worked top-2 router calculation

Suppose one token reaches a layer with four experts. Its router logits are 1.2, 0.2, -0.4, and 0.8.

Top-2 selection keeps expert 1 at 1.2 and expert 4 at 0.8. Experts 2 and 3 do not run for this token.

Apply softmax over the two kept logits. Their exponentials are approximately 3.320 and 2.226.

The selected sum is 5.546. Expert 1 receives weight 3.320 / 5.546, or about 0.599. Expert 4 receives about 0.401.

Now use scalar outputs only to make the final step visible. Suppose expert 1 returns 2.0 and expert 4 returns -1.0 for one output coordinate.

The mixed coordinate is 0.599 times 2.0 plus 0.401 times -1.0. The result is about 0.797.

Real experts return vectors, so the same weighted addition happens at every coordinate.

The Mixtral report uses eight experts in each MoE layer and selects two per token. Its router applies softmax to the top-k logits.

Top-k selection is discrete. Gradients still train the selected gate weights and experts, while auxiliary objectives or other balancing methods shape which experts receive tokens.

Sparse compute does not mean sparse storage

Selecting two of eight experts avoids executing all eight feed-forward networks for one token.

The deployment still stores every expert's weights. Mixtral has about 47 billion total parameters but uses about 13 billion active parameters for each token.

Total parameters determine the memory needed to host the model. Active parameters more closely describe arithmetic per token, though routing and communication add work.

This is why an MoE can need several GPUs even when its active parameter count resembles a smaller dense model.

Fanout's tensor versus pipeline parallelism guide explains how hardware boundaries change communication cost.

MoE adds another layout problem called expert parallelism. Different devices own different experts, so tokens must travel to the devices selected by their routers.

After expert computation, outputs travel back to the token's original sequence layout. NVIDIA's MoE systems overview traces the dispatch and aggregation steps across devices.

The router's mathematical cost is small. The data movement caused by its choices may not be.

Expert capacity turns scores into queues

The Switch Transformer paper treats expert capacity, routing cost, and training stability as parts of the same design. One top-2 decision creates two assignments.

A batch of T tokens therefore creates 2T assignments.

Suppose a batch contains 12 tokens, four experts, and top-2 routing. There are 24 assignments, so a perfectly even load is six assignments per expert.

Set the capacity factor to 1.25. A common capacity calculation gives ceiling(1.25 times 24 / 4), which is eight token slots per expert.

If expert 1 receives nine assignments, one does not fit its eight-slot buffer.

Older capacity-limited designs can drop or bypass overflow assignments through the residual path. Other systems reroute tokens, pad buffers, or use dropless kernels with variable work.

Switch Transformer simplified routing to one expert per token. That cuts the number of expert assignments in half relative to top-2 routing for the same token batch.

A larger capacity factor reduces overflow risk but reserves more buffer space and can waste computation through padding.

A smaller factor improves the best-case packing efficiency but leaves less room for a popular expert.

Capacity is both a model hyperparameter and a systems constraint. It fixes array shapes and controls how much imbalance the runtime can absorb.

Why routers need load balancing

Without pressure to spread tokens, a few experts can become popular early in training.

Those experts then receive more examples and improve faster. The router may favor them even more, while other experts receive too little training.

The original sparse MoE work added noisy top-k gating and balancing terms. Switch Transformer used an auxiliary loss that encourages more even routing.

Balancing does not require every token type to visit every expert equally. It tries to prevent batch-level demand from collapsing onto a small part of the expert pool.

Some newer models change the mechanism. Expert-choice routing lets each expert select tokens up to a fixed capacity rather than letting each token select experts independently.

Google's expert-choice routing description explains the trade: expert loads become even, but tokens can receive a variable number of experts.

Other models adjust selection biases without adding a large auxiliary loss. The router family must be named before comparing load-balance claims.

Measure assignment counts, overflow rate, expert utilization, and the slowest expert for each layer. An average across all layers can hide one recurring hotspot.

Experts are not clean subject specialists

The word "expert" invites a misleading picture in which one network learns mathematics and another learns French.

Some routing patterns are interpretable, but specialization can follow tokens, syntax, position, or other internal features rather than human topic categories.

The Mixtral report found no clear expert assignment pattern by broad topic across its tested domains. It did find local structure, including consecutive tokens often choosing the same experts.

The safe description is functional specialization learned from the training objective. A router score is not an explanation of why the model produced an answer.

Shared experts make the picture less tidy again. They process every token while routed experts provide sparse additional capacity.

The official Kimi K3 model summary lists 896 experts, 16 selected experts per token, and two shared experts.

Fanout's Kimi K3 architecture guide connects that expert layout to the model's other attention and residual changes.

A model with shared experts spends some dense feed-forward compute on every token. Counting only routed top-k experts understates its active path.

Routing changes inference topology

Expert parallelism places different expert weights on different GPUs. A balanced arithmetic count can still become a network bottleneck if assignments cross slow links.

The serving trace has three parts: token dispatch, expert computation, and output aggregation.

For each layer, inspect which ranks own the selected experts and how many bytes the dispatch moves.

Co-locating frequently used experts can reduce transfers but may create memory imbalance. Replicating popular experts uses more memory and complicates routing consistency.

Large batches give each expert more tokens and can make its matrix multiplication efficient. They also increase queueing and the size of all-to-all exchanges.

Continuous batching changes the active token set between iterations. Fanout's continuous batching guide shows why the assignment mix can change as requests arrive and finish.

For serving, record router time, dispatch time, expert kernel time, combine time, and idle time by rank.

Also compare one-request latency with loaded throughput. An MoE can have low active FLOPs yet deliver poor latency when small expert batches and communication dominate.

Read an MoE specification in this order

Start with the number of routed experts, shared experts, and selected experts per token.

Then identify whether the router uses softmax or sigmoid scores, top-k token choice or expert choice, and whether selected weights are renormalized.

Check the capacity formula, overflow behavior, balancing objective, and whether training and inference use different settings.

For deployment, map expert ownership to nodes and links. Confirm the dispatch primitive, supported precision, expert kernel, and maximum tokens per expert.

Finally, trace distributions rather than one illustrative token. Record how routing changes by layer, prompt type, sequence position, and concurrency.

The one-token calculation explains the model. The batch distribution explains whether the system can run it efficiently.