---
title: "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity"
description: "Switch Transformers shows how to grow a model's capacity without activating every parameter for every token. Read it to see how top-1 expert routing, load balancing, and expert capacity turn mixture-of-experts into a simpler scaling recipe while keeping per-token computation roughly constant."
canonical_url: "https://fanout.sh/daily/2026-07-30-switch-transformers"
md_url: "https://fanout.sh/daily/2026-07-30-switch-transformers.md"
last_updated: "2026-07-30"
access: "public"
---

# Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Switch Transformers shows how to grow a model's capacity without activating every parameter for every token. Read it to see how top-1 expert routing, load balancing, and expert capacity turn mixture-of-experts into a simpler scaling recipe while keeping per-token computation roughly constant.

## Paper details

- Authors: Fedus, Zoph, and Shazeer

- Venue: JMLR 2022

- Track: AI Research

- Difficulty: Medium

- Reading time: 50 min

- Original paper: https://www.jmlr.org/papers/v23/21-0998.html

## What you will learn

- How a Switch layer routes each token to one expert feed-forward network instead of evaluating every expert.

- Why an auxiliary load-balancing loss and a fixed expert capacity keep routing from overloading a small set of experts.

- How sparse activation adds model capacity at roughly fixed computation per token, producing large pre-training speedups over dense T5 baselines.

## Continue exploring

[Explore Mixture of Experts](https://fanout.sh/knowledge-graph?node=ai-research%3Aconcept%3Amixture-of-experts-moe): Open Fanout's mixture-of-experts node and connect the paper to sparse activation, token routing, expert capacity, load balancing, and efficient model scaling.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md