FlashAttention

FlashAttention keeps exact attention but changes the memory traffic pattern. It is a clean example of why practical model speed often comes from respecting hardware limits rather than changing the math.

Reading focus: Why attention can be IO-bound even when the formula looks simple. How tiling and recomputation reduce high-bandwidth memory reads and writes. Why kernel-level details matter for serving long-context models.

NeurIPS 2022. Dao et al.. 55 min read, hard difficulty.