Top-k mixture-of-experts routing

Makes sparse conditional computation explicit by separating expert selection from the weighted combination of only selected expert outputs.

The selected expert set T of x is top k of router probabilities p of x, and output y is the sum over selected experts of normalized routing weight p tilde e times expert E e of x.

Choose a few experts for each token, run only those experts, and blend their answers using router weights.

Top-k creates sparse control flow, and the following summation combines only the active branches.

Expert capacity, token dispatch, all-to-all communication, and load-balancing losses determine whether this mathematical sparsity produces efficient distributed execution.