Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Megatron-LM places tensor boundaries around the operations that can remain local. Follow why a naive row split must synchronize before GeLU, how paired column- and row-parallel projections reduce only at the block output, and what the paper's 512-GPU weak-scaling evidence actually establishes.

Reading focus: Why partial sums must be combined before a nonlinearity, while disjoint output columns can pass through GeLU independently. How the same column-then-row arrangement keeps complete attention heads local and limits collectives per Transformer layer. Why 74% weak-scaling efficiency is not a 74% fixed-model speedup, and how interconnect and GEMM shape bound the result.

arXiv 2019 · revised 2020. Mohammad Shoeybi et al.. 12 min read, hard difficulty.