---
title: "Row-parallel tensor communication"
description: "Shows why sharding a linear layer across its input dimension creates full-shaped partial outputs that require a sum collective."
canonical_url: "https://fanout.sh/labs/math-decoder/formula/row-parallel-linear-all-reduce"
md_url: "https://fanout.sh/labs/math-decoder/formula/row-parallel-linear-all-reduce.md"
last_updated: "2026-08-09"
access: "public"
---

# Row-parallel tensor communication

Shows why sharding a linear layer across its input dimension creates full-shaped partial outputs that require a sum collective.

## Public overview

Each rank r computes local output Y r as input shard X r times weight shard W r, then global Y is the sum all-reduce of all rank-local partial outputs from rank one through p.

Every device computes part of each output activation, then the devices add those parts together.

Sum all-reduce is the distributed reduction that reconstructs the mathematical result of the unsharded matrix product.

Column-parallel layers instead produce distinct output shards; Megatron pairs row and column partitions to avoid unnecessary collectives between adjacent operations.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md