FlashAttention

A calculation that runs one tile at a time needs somewhere to put the tile. The tiled attention and online softmax lesson freed attention's softmax from needing a whole row before it could start, and that freed the whole computation to walk over small pieces of the inputs while carrying a running maximum, a running normalizer and a partial output.…

In this lesson

• Fast memory and slow memory
• The IO complexity claim
• Where the tiles live
• What inference inherits
• FlashAttention-2 and work partitioning

Continue learning

Pro unlocks the complete lesson, video, practice, and code where available.