GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
GShard made giant sparse Transformers programmable by pairing top-2 token routing with capacity limits and compiler-propagated sharding. Follow the router, expert dispatch, all-to-all communication, SPMD annotations, multilingual translation evidence, and the limits hidden behind the 600-billion-parameter headline.
Reading focus: How top-2 gating activates only a small expert subset for each token while the rest of the model remains shared. Why expert capacity, stochastic second-expert routing, and an auxiliary load-balancing loss are systems controls as much as modeling choices. How a few split and replicate annotations let an SPMD compiler infer expert sharding, reshard tensors, and generate one program for every device.
ICLR 2021. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 11 min read, hard difficulty.