---
title: "GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism"
description: "GPipe divides a neural network into sequential stages and sends micro-batches through them as a pipeline. Read it to see how the schedule trades idle time for throughput, why accumulated gradients preserve synchronous updates, and how rematerialization makes much larger models fit in accelerator memory."
canonical_url: "https://fanout.sh/daily/2026-10-11-gpipe-pipeline-parallelism"
md_url: "https://fanout.sh/daily/2026-10-11-gpipe-pipeline-parallelism.md"
last_updated: "2026-10-11"
access: "public"
---

# GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism

GPipe divides a neural network into sequential stages and sends micro-batches through them as a pipeline. Read it to see how the schedule trades idle time for throughput, why accumulated gradients preserve synchronous updates, and how rematerialization makes much larger models fit in accelerator memory.

## Paper details

- Authors: Yanping Huang et al.

- Venue: NeurIPS 2019

- Track: Systems

- Difficulty: Easy

- Reading time: 32 min

- Original paper: https://arxiv.org/abs/1811.06965v5

## What you will learn

- How layer partitions and micro-batches turn model parallelism into a diagonal pipeline schedule.

- Why increasing the number of micro-batches shrinks the fill-and-drain bubble without changing the mini-batch update.

- How activation rematerialization saves memory and why stage balance still limits throughput.

## Continue exploring

[Map distributed training](https://fanout.sh/knowledge-graph?node=ai-research%3Aconcept%3Adistributed-training): Connect pipeline parallelism to data, tensor, and optimizer-state partitioning in the Fanout knowledge map.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md