FlashAttention

Attention isn't compute-bound — it's memory-bandwidth-bound. FlashAttention restructures the computation into tiles that live in fast on-chip SRAM, never materializing the full attention matrix in slow HBM. Exact same math, 2-4× faster and linear memory, which is why it's in every serious training and inference stack.