Switch Transformers: Scaling to Trillion…

Switch Transformers shows how to grow a model's capacity without activating every parameter for every token. Read it to see how top-1 expert routing, load balancing, and expert capacity turn mixture-of-experts into a simpler scaling recipe while keeping per-token computation roughly constant.

Reading focus: How a Switch layer routes each token to one expert feed-forward network instead of evaluating every expert. Why an auxiliary load-balancing loss and a fixed expert capacity keep routing from overloading a small set of experts. How sparse activation adds model capacity at roughly fixed computation per token, producing large pre-training speedups over dense T5 baselines.

JMLR 2022. Fedus, Zoph, and Shazeer. 50 min read, medium difficulty.