---
title: "GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding"
description: "GShard made giant sparse Transformers programmable by pairing top-2 token routing with capacity limits and compiler-propagated sharding. Follow the router, expert dispatch, all-to-all communication, SPMD annotations, multilingual translation evidence, and the limits hidden behind the 600-billion-parameter headline."
canonical_url: "https://fanout.sh/daily/2026-11-18-gshard-conditional-computation"
md_url: "https://fanout.sh/daily/2026-11-18-gshard-conditional-computation.md"
last_updated: "2026-11-18"
access: "public"
---

# GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

GShard made giant sparse Transformers programmable by pairing top-2 token routing with capacity limits and compiler-propagated sharding. Follow the router, expert dispatch, all-to-all communication, SPMD annotations, multilingual translation evidence, and the limits hidden behind the 600-billion-parameter headline.

## Paper details

- Authors: Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen

- Venue: ICLR 2021

- Track: Systems

- Difficulty: Hard

- Reading time: 11 min

- Original paper: https://arxiv.org/abs/2006.16668

## What you will learn

- How top-2 gating activates only a small expert subset for each token while the rest of the model remains shared.

- Why expert capacity, stochastic second-expert routing, and an auxiliary load-balancing loss are systems controls as much as modeling choices.

- How a few split and replicate annotations let an SPMD compiler infer expert sharding, reshard tensors, and generate one program for every device.

## Continue exploring

[Compare Switch Transformers](https://fanout.sh/daily/2026-07-30-switch-transformers): Compare GShard's top-2 routing and automatic sharding with the later top-1 Switch recipe that simplified sparse expert computation.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md
