---
title: "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models"
description: "ZeRO asks why adding data-parallel workers does not make a model easier to fit when every worker keeps the same training state. Read it to see how partitioning optimizer states, gradients, and parameters turns aggregate cluster memory into usable model capacity while preserving the data-parallel computation."
canonical_url: "https://fanout.sh/daily/2026-08-05-zero-memory-optimization"
md_url: "https://fanout.sh/daily/2026-08-05-zero-memory-optimization.md"
last_updated: "2026-08-05"
access: "public"
---

# ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

ZeRO asks why adding data-parallel workers does not make a model easier to fit when every worker keeps the same training state. Read it to see how partitioning optimizer states, gradients, and parameters turns aggregate cluster memory into usable model capacity while preserving the data-parallel computation.

## Paper details

- Authors: Rajbhandari et al.

- Venue: SC 2020

- Track: Systems

- Difficulty: Medium

- Reading time: 50 min

- Original paper: https://arxiv.org/abs/1910.02054v3

## What you will learn

- Why mixed-precision Adam needs 16 bytes of persistent model state per parameter on every conventional data-parallel worker.

- How ZeRO's three cumulative stages shard optimizer states, gradients, and parameters across the data-parallel group.

- Why Stages 1 and 2 keep the paper's baseline communication volume while full parameter partitioning raises it to 1.5 times baseline.

## Continue exploring

[Explore Memory Optimization](https://fanout.sh/ai/hardware): Continue into Fanout's hardware guide for data parallelism, DeepSpeed ZeRO, mixed precision, interconnects, activation checkpointing, and GPU memory planning.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md