---
title: "FlashAttention"
description: "How FlashAttention keeps attention tiles in fast on-chip memory, its IO complexity claim, what inference inherits, and FlashAttention-2's partitioning."
canonical_url: "https://fanout.sh/inference-eng/curriculum/14-03"
md_url: "https://fanout.sh/inference-eng/curriculum/14-03.md"
access: "pro-preview"
---

# FlashAttention

How FlashAttention keeps attention tiles in fast on-chip memory, its IO complexity claim, what inference inherits, and FlashAttention-2's partitioning.

- Access: Fanout Pro (public preview)

## Public preview

A calculation that runs one tile at a time needs somewhere to put the tile. The tiled attention and online softmax lesson freed attention's softmax from needing a whole row before it could start, and that freed the whole computation to walk over small pieces of the inputs while carrying a running maximum, a running normalizer and a partial output.…

## In this lesson

- Fast memory and slow memory
- The IO complexity claim
- Where the tiles live
- What inference inherits
- FlashAttention-2 and work partitioning

## Continue learning

Pro unlocks the complete lesson, video, practice, and code where available.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md