FlashAttention vs PagedAttention: what each fixes

FlashAttention vs PagedAttention reads like a choice between two competing attention optimizations. It is not. The two remove different bottlenecks, and every major serving engine runs both at the same time.

FlashAttention changes how the attention kernel moves activations through GPU memory while it computes. PagedAttention changes how the KV cache is allocated between steps. One is compute-path IO, the other is memory management.

The useful question is which bottleneck is currently limiting your workload. That depends on request phase and concurrency, and each side has numbers.

Two different problems inside one layer

Standard attention materializes an N by N score matrix per head. At a 4,096-token sequence that is 16.8 million scores, written to and read back from HBM, the GPU's large but slow memory.

That matrix is transient. It exists only inside one layer's forward pass, and its cost is the traffic between HBM and the small on-chip SRAM where the arithmetic happens.

The KV cache is the opposite: persistent state. It grows by one entry per generated token per layer and lives for the entire request. Fanout's KV cache formula guide works through its size.

FlashAttention attacks the first problem. PagedAttention attacks the second. Neither touches the other's territory.

What FlashAttention actually changes

The FlashAttention paper computes attention in SRAM-sized tiles with an online softmax, so the full score matrix never lands in HBM. Memory use drops from quadratic to linear in sequence length.

The gap it exploits is physical. An A100 reads HBM at about 2 TB per second, while its on-chip SRAM moves roughly 19 TB per second. Keeping tiles in SRAM turns a memory-bound kernel into a compute-bound one.

The result is exact attention, not an approximation, delivered faster: a 3x speedup on GPT-2 at 1K tokens, 15 percent end to end on BERT-large, and 2.4x on long-range benchmarks.

FlashAttention-2 rewrote the kernel for roughly another 2x, and FlashAttention-3 targets Hopper GPUs. Each generation improves the same quantity: bytes moved per unit of attention math.

There is a serving subtlety. The original win shrinks during decode, because a single-token query has no large score matrix to avoid. Version 2.2 added kernels tuned for exactly that shape.

Fanout's FlashAttention explainer walks the tiling and online softmax mechanics in detail.

What PagedAttention actually changes

The vLLM paper measured prior serving systems wasting 60 to 80 percent of KV cache memory through fragmentation and reserved-but-unused slots.

The stakes scale with the model. For OPT-13B, one token's keys and values take about 800 KB across 40 layers, so a 2,048-token request holds 1.6 GB of cache. Reserving that for every slot, filled or not, is how waste reached 60 percent.

Requests were handed contiguous slabs sized for a maximum length most never reached. PagedAttention splits the cache into fixed-size blocks and maps them through a per-request block table, the way an operating system pages virtual memory.

Waste falls below 4 percent. The reclaimed memory becomes batch slots, and vLLM reported 2 to 4x the throughput of FasterTransformer and Orca at the same latency.

Note what did not change: the attention arithmetic. The paper is explicit that PagedAttention alters memory layout and allocation, not the math. The speedup is more concurrent requests, not a faster kernel.

Blocks also enable sharing. Parallel sampling and beam search reuse prompt blocks copy-on-write instead of duplicating them. Fanout's PagedAttention guide covers the block table mechanics.

They compose, and engines ship both

Because the two operate on different objects, they stack. vLLM runs FlashAttention-class kernels that read the paged cache directly, and FlashAttention itself added a block table argument for paged KV in version 2.5.

Composition has a price, though. A kernel that chases a block table is harder to write and slightly slower than one reading contiguous memory.

The vAttention paper makes that case. It keeps the cache contiguous in virtual memory using CUDA allocation APIs and reports up to 1.23x higher throughput than paged FlashAttention and FlashInfer kernels.

vAttention is the exception that clarifies the rule: paging is a memory-management trade, worth it when fragmentation is the constraint, not a free win.

A request-phase decision guide

Prefill is compute-bound. The whole prompt is processed at once, score matrices are large, and kernel efficiency decides latency. If your workload is long prompts at low concurrency, FlashAttention-class kernels are the lever.

Decode is different. Each step attends one query against the whole cache, so there is no large score matrix to avoid. The step cost is streaming KV bytes, which is bandwidth, not kernel cleverness.

What helps decode is anything that shrinks KV bytes per step: grouped-query attention, quantized caches, shorter contexts. Those are different levers from either technique here.

Fanout's prefill and decode guide covers why the two phases hit different hardware limits.

Concurrency is where PagedAttention earns its keep. Batch slots are bought with KV memory, and continuous batching turns reclaimed bytes into served requests.

A serving cluster at high occupancy with variable output lengths is PagedAttention territory. A single user running one long document through a local model barely touches it.

A useful proxy: watch the engine's cache usage and preemption counters. If blocks run out while compute sits idle, memory management is the constraint. If compute saturates during prompt processing, the kernel is.

FlashAttention vs PagedAttention, in short

FlashAttention makes each attention call cheaper by never materializing the score matrix in HBM. PagedAttention makes more calls fit by allocating the KV cache in blocks instead of contiguous slabs.

They are complements. The real decision is which bottleneck to tune first.

• Long prompts and few streams: profile the attention kernel first, since prefill compute dominates.
• Many concurrent chats with variable output lengths: measure KV waste and batch occupancy first.
• Decode-heavy at batch one: neither is the main lever, since weight and KV bandwidth dominate the step time.
• Already on vLLM, TensorRT-LLM, or SGLang: you run both today, so tune block size and kernel backend rather than choosing.

The versus comes from how people search, and the engines answered it years ago by shipping both.