--- title: "Why is FlashAttention faster?" description: "FlashAttention is faster because it avoids writing full attention matrices to HBM. Follow the 4K-token memory traffic and exact softmax method." canonical_url: "https://fanout.sh/blog/why-is-flash-attention-faster" md_url: "https://fanout.sh/blog/why-is-flash-attention-faster.md" last_updated: "2026-08-13" access: "public" --- # Why is FlashAttention faster? FlashAttention is faster because it avoids writing full attention matrices to HBM. Follow the 4K-token memory traffic and exact softmax method. - Author: Suraj Gaud - Published: 2026-08-13 - Track: Inference engineering - Tags: FlashAttention, attention, GPU memory, HBM, online softmax, LLM inference Why is FlashAttention faster? It moves far less data between GPU high-bandwidth memory and fast on-chip memory. It still computes standard softmax attention. The speed comes from tiling, online softmax, and kernel fusion, which avoid writing the full token-by-token attention matrices to HBM. The arithmetic count stays quadratic in sequence length. The expensive memory traffic drops because intermediate scores and probabilities remain in small on-chip tiles. ## Why is FlashAttention faster on long sequences? For one attention head, standard attention computes the score matrix S as Q times K transposed. It then computes P as softmax of S and the output O as P times V. The[original FlashAttention paper](https://arxiv.org/abs/2205.14135)points out that a conventional implementation runs these as separate stages and materializes S and P in HBM. HBM is large and fast compared with host memory, but it sits farther from the arithmetic units than registers and on-chip SRAM. Moving a large intermediate out and back can take longer than the arithmetic applied to it. Standard attention writes S, reads S for softmax, writes P, then reads P for the value multiplication. Masking and dropout can add more passes or intermediates. The matrix size grows with the square of sequence length. Q, K, V, and O grow with sequence length times head dimension, but S and P grow with sequence length times sequence length. Once those square matrices dominate, optimizing only the matrix multiplications leaves the memory bottleneck in place. ## The 4K-token traffic FlashAttention avoids Take one head with 4,096 tokens, head dimension 128, and BF16 storage. Q, K, or V contains 4,096 times 128 elements, so each tensor is 1 MiB. The score matrix contains 4,096 times 4,096 elements. At two bytes each, S is 32 MiB. The probability matrix P is another 32 MiB. Ignoring every other transfer, writing and rereading S moves 64 MiB. Writing and rereading P moves another 64 MiB. That is at least 128 MiB of HBM traffic per head just for the two square intermediates. With 32 heads, it becomes 4 GiB for one attention layer. The number is an illustration of a dense materialized pipeline, not a benchmark for every framework. Fused kernels, causal layouts, data types, and recomputation policies change the exact traffic. The scaling remains the problem. Doubling the sequence from 4K to 8K makes each square matrix four times larger. FlashAttention never stores a complete S or P matrix in HBM. It forms a small score tile on chip, uses it to update the output accumulator, and discards that tile. That removes the 128 MiB term in the worked example. It still reads input tiles and may reread them across blocks, so its HBM traffic is not zero. ## Tiling keeps intermediates close to compute FlashAttention splits Q into row blocks and K and V into column blocks sized to fit the GPU's on-chip memory. For one Q block, the kernel loads a K and V block, computes a small score tile, applies the partial softmax update, and accumulates a partial output. The[Stanford explanation](https://hazyresearch.stanford.edu/blog/2023-01-12-flashattention-long-sequences)describes this movement from HBM to SRAM and the parallel assignment of attention rows to thread blocks. After all K and V blocks have contributed, the kernel writes the completed output block to HBM. It does not preserve the score tiles. Tiling by itself is common in matrix multiplication. Attention is harder because softmax for one query depends on every score in that query's row. A tile does not initially know the maximum score or denominator from later tiles. FlashAttention needs an update rule that can combine partial softmax results without retaining the whole row. ## Online softmax keeps the answer exact For each query row, the kernel keeps a running maximum, a running exponential sum, and a running weighted-value accumulator. When a new score tile has a larger maximum, the kernel rescales the previous sum and accumulator into the new numerical frame. It then adds the new tile's contribution. After the last tile, dividing the accumulated weighted values by the accumulated denominator gives the same softmax attention result in real arithmetic. The[FlashAttention proof](https://papers.neurips.cc/paper_files/paper/2022/file/67d57c32e20fd0a7a302cb81d36e40d5-Paper-Conference.pdf)develops this blockwise recurrence and proves the algorithm returns softmax of QK transposed times V. "Exact" means FlashAttention does not approximate attention by dropping tokens, projecting features, or replacing softmax. It does not promise bit-for-bit identity with every reference kernel. Floating-point additions occur in a different order, so the last bits can differ within numerical tolerance. The[official implementation](https://github.com/Dao-AILab/flash-attention)tests outputs and gradients against a reference and bounds the error relative to the baseline implementation. ## Extra computation can still finish sooner The backward pass normally needs the probability matrix. Saving that square matrix would restore the memory cost FlashAttention removed. Instead, FlashAttention saves the output and compact softmax statistics, then recomputes score and probability tiles during backward. Recomputation adds arithmetic. It can still reduce runtime because arithmetic units are faster than the avoided HBM round trips for this memory-bound operation. The original paper reports one A100 experiment with 66.6 GFLOPs for standard attention and 75.2 GFLOPs for FlashAttention. In the same experiment, HBM reads and writes fell from 35.3 GB to 4.4 GB, while forward-plus-backward runtime fell from 35.1 ms to 11.7 ms. Those numbers used sequence length 1,024, head dimension 64, 16 heads, batch size 64, a key-padding mask, and no dropout. They show the mechanism, not a universal speedup. The paper expresses standard attention traffic as Theta of N times d plus N squared. FlashAttention uses Theta of N squared times d squared divided by on-chip memory size M. For typical head dimensions and SRAM sizes, d squared is much smaller than M. Larger useful tiles mean fewer passes over inputs and less HBM traffic. ## Kernel fusion removes stage boundaries A high-level attention expression looks like three operations. A conventional implementation may launch separate kernels for matrix multiplication, masking, softmax, dropout, and the value multiplication. Each boundary can require an intermediate to be written where the next kernel can read it. FlashAttention fuses the attention path into one GPU kernel. Score tiles, normalization state, and output accumulators stay in registers or shared memory while that kernel runs. Fusion and tiling depend on each other. Fusion keeps the tile live across operations, while tiling makes the working set small enough to remain on chip. Later versions improve the work partition rather than changing the basic attention definition. The[FlashAttention-2 paper](https://arxiv.org/abs/2307.08691)reduces non-matrix arithmetic and improves parallelism across GPU thread blocks and warps. This is why version labels matter in benchmarks. A result for FlashAttention-2, FlashAttention-3, or a framework's fused attention backend is not automatically a result for the original kernel. ## Why FlashAttention is not always faster At short sequence lengths, the square intermediates are small. Kernel launch overhead, tile setup, and hardware occupancy can matter more than avoided HBM traffic. Head dimension, data type, causal masking, dropout, GPU architecture, and backward requirements determine which kernel is supported and which tile sizes work well. The official repository currently lists distinct CUDA and ROCm support boundaries. It also shows different speedups by sequence length and GPU memory bandwidth. Framework dispatch can hide this choice. Current[PyTorch scaled dot product attention documentation](https://docs.pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention)can select among FlashAttention and other backends. Confirm the selected backend instead of assuming a function name proves which kernel ran. Then benchmark the actual shapes and masks used by the model. A good test sweeps sequence length, batch size, head dimension, causal mode, and data type. Record attention time, peak memory, and end-to-end model time. An attention kernel can be much faster while total model latency changes modestly if feed-forward layers, communication, sampling, or scheduler overhead dominate. ## FlashAttention is not KV-cache compression FlashAttention removes the need to materialize square attention intermediates. It does not remove the persistent keys and values needed for autoregressive decoding. During training and prompt prefill, many query tokens attend at once, so the square matrix and activation-memory savings are substantial. During one-token decode, the new query length is one. The operation reads the existing KV cache, but it does not create an N by N score matrix for N new queries. The[prefill vs decode guide](/blog/prefill-vs-decode-llm-inference)traces this difference across one request. Long-context decode can remain limited by reading the cache. The[KV cache memory guide](/blog/kv-cache-formula-llm-inference-memory)calculates that persistent storage separately. Paged KV allocation, cache quantization, continuous batching, and FlashAttention solve different parts of the serving path. They can coexist without being interchangeable. ## Read the profiler as memory traffic To verify why FlashAttention is faster for a workload, compare the selected kernels under identical inputs and inspect more than elapsed time. Measure: - HBM bytes read and written by the attention region. - Peak activation memory during forward and backward. - Kernel count and gaps between launches. - Tensor-core utilization and achieved memory bandwidth. - Runtime by sequence length, head dimension, batch size, and mask type. - End-to-end model latency and throughput. The expected signature is fewer HBM transfers, no stored N by N probability matrix, and a lower peak-memory curve as the sequence grows. FlashAttention is faster when the avoided data movement costs more than tiling, online normalization, and any recomputation. The win comes from organizing exact attention around the GPU memory hierarchy. --- This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included. Browse the public content map: https://fanout.sh/sitemap.md