---
title: "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning"
description: "FlashAttention-2 asks why an exact, I/O-aware attention kernel can still leave much of an A100 idle. Read it to see how reducing non-matmul work, parallelizing along sequence tiles, and giving each warp disjoint query and output rows improves GPU utilization without changing the attention result."
canonical_url: "https://fanout.sh/daily/2026-08-08-flashattention-2"
md_url: "https://fanout.sh/daily/2026-08-08-flashattention-2.md"
last_updated: "2026-08-08"
access: "public"
---

# FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

FlashAttention-2 asks why an exact, I/O-aware attention kernel can still leave much of an A100 idle. Read it to see how reducing non-matmul work, parallelizing along sequence tiles, and giving each warp disjoint query and output rows improves GPU utilization without changing the attention result.

## Paper details

- Authors: Tri Dao

- Venue: ICLR 2024

- Track: Inference

- Difficulty: Hard

- Reading time: 50 min

- Original paper: https://arxiv.org/abs/2307.08691v1

## What you will learn

- Why scalar softmax bookkeeping and synchronization are expensive relative to tensor-core matrix multiplication on the paper's A100 setup.

- How sequence-level thread-block parallelism supplies more independent work when batch size and head count are too small to occupy the GPU.

- Why splitting query rows across warps while sharing key and value tiles removes the cross-warp reduction of partial output rows.

## Continue exploring

[Explore FlashAttention](https://fanout.sh/knowledge-graph?node=ai-research%3Aconcept%3Aflash-attention): Open Fanout's FlashAttention node and connect exact attention tiling to GPU memory hierarchy, fused kernels, sequence length, and modern inference systems.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md