---
title: "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism"
description: "Megatron-LM places tensor boundaries around the operations that can remain local. Follow why a naive row split must synchronize before GeLU, how paired column- and row-parallel projections reduce only at the block output, and what the paper's 512-GPU weak-scaling evidence actually establishes."
canonical_url: "https://fanout.sh/daily/2026-10-27-megatron-lm-tensor-parallelism"
md_url: "https://fanout.sh/daily/2026-10-27-megatron-lm-tensor-parallelism.md"
last_updated: "2026-10-27"
access: "public"
---

# Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Megatron-LM places tensor boundaries around the operations that can remain local. Follow why a naive row split must synchronize before GeLU, how paired column- and row-parallel projections reduce only at the block output, and what the paper's 512-GPU weak-scaling evidence actually establishes.

## Paper details

- Authors: Mohammad Shoeybi et al.

- Venue: arXiv 2019 · revised 2020

- Track: Systems

- Difficulty: Hard

- Reading time: 12 min

- Original paper: https://arxiv.org/abs/1909.08053v4

## What you will learn

- Why partial sums must be combined before a nonlinearity, while disjoint output columns can pass through GeLU independently.

- How the same column-then-row arrangement keeps complete attention heads local and limits collectives per Transformer layer.

- Why 74% weak-scaling efficiency is not a 74% fixed-model speedup, and how interconnect and GEMM shape bound the result.

## Continue exploring

[Compare pipeline parallelism](https://fanout.sh/daily/2026-10-11-gpipe-pipeline-parallelism): Contrast splitting individual matrices inside a layer with GPipe's strategy of assigning whole layer groups to pipeline stages.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md