---
title: "Tensor Parallelism"
description: "How several GPUs work on the same tokens: output slices and partial sums, a two-worker example, Llama's feed-forward block, and attention head limits."
canonical_url: "https://fanout.sh/inference-eng/curriculum/18-03"
md_url: "https://fanout.sh/inference-eng/curriculum/18-03.md"
access: "pro-preview"
---

# Tensor Parallelism

How several GPUs work on the same tokens: output slices and partial sums, a two-worker example, Llama's feed-forward block, and attention head limits.

- Access: Fanout Pro (public preview)

## Public preview

A large layer can be divided so that several GPUs work on the same tokens at the same time. Each holds a different slice of the layer's weights. The useful question is where those slices can compute independently and where their results must meet before the model can continue.

## In this lesson

- Output slices and partial sums
- A two-worker calculation
- Llama's feed-forward block
- Attention heads and partition limits
- The next operation's dependency

## Continue learning

Pro unlocks the complete lesson, video, practice, and code where available.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md