ZeRO: Memory Optimizations Toward Training…

ZeRO asks why adding data-parallel workers does not make a model easier to fit when every worker keeps the same training state. Read it to see how partitioning optimizer states, gradients, and parameters turns aggregate cluster memory into usable model capacity while preserving the data-parallel computation.

Reading focus: Why mixed-precision Adam needs 16 bytes of persistent model state per parameter on every conventional data-parallel worker. How ZeRO's three cumulative stages shard optimizer states, gradients, and parameters across the data-parallel group. Why Stages 1 and 2 keep the paper's baseline communication volume while full parameter partitioning raises it to 1.5 times baseline.

SC 2020. Rajbhandari et al.. 50 min read, medium difficulty.