---
title: "Multi-head attention shape flow"
description: "Makes the attention equation type-checkable by exposing every tensor axis and the contracted dimensions."
canonical_url: "https://fanout.sh/labs/math-decoder/formula/multi-head-attention-tensor-shapes"
md_url: "https://fanout.sh/labs/math-decoder/formula/multi-head-attention-tensor-shapes.md"
last_updated: "2026-08-09"
access: "public"
---

# Multi-head attention shape flow

Makes the attention equation type-checkable by exposing every tensor axis and the contracted dimensions.

## Public overview

Q K and V have batch, head, token, and head-width axes; attention weights have batch, head, query-token, and key-token axes; multiplying by V restores a final head-width axis.

Within every batch and head, compare every query token with every key token, then use those weights to combine value features.

Tensor contraction is the structural verb; each matrix product removes one named shared axis and preserves the others.

Real implementations may use different query and key lengths, grouped KV heads, packed sequences, or fused layouts while preserving the same contraction logic.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md