--- title: "vAttention vs PagedAttention for KV cache" description: "Compare CUDA virtual memory with software KV blocks, including kernel overhead, page granularity, prefix sharing, and a practical choice rule." canonical_url: "https://fanout.sh/blog/vattention-vs-pagedattention" md_url: "https://fanout.sh/blog/vattention-vs-pagedattention.md" last_updated: "2026-08-22" access: "public" --- # vAttention vs PagedAttention for KV cache Compare CUDA virtual memory with software KV blocks, including kernel overhead, page granularity, prefix sharing, and a practical choice rule. - Author: Suraj Gaud - Published: 2026-08-22 - Track: Inference engineering - Access: Public - Tags: vAttention, PagedAttention, KV cache, CUDA, virtual memory, vLLM, FlashAttention, LLM inference vAttention vs PagedAttention is a choice about who translates a token position into physical GPU memory. PagedAttention does it in the serving engine and attention kernel. vAttention asks CUDA virtual memory to do it. Both allocate KV cache memory as a sequence grows. Both avoid reserving every request's maximum context length. The difference is whether the attention kernel sees blocks or one contiguous virtual tensor. That difference changes kernel portability, lookup overhead, allocation granularity, prefix sharing, and operational maturity. ## vAttention vs PagedAttention PagedAttention divides a request's KV cache into logical blocks. A block table maps each logical block to a physical GPU block, which can live anywhere in the cache pool. The[PagedAttention paper](https://arxiv.org/abs/2309.06180)compares that design to operating-system paging. A request receives new physical blocks as its sequence grows and releases them when it finishes. The attention kernel reads the block table and gathers K and V from non-contiguous locations. It must understand the block layout. vAttention reserves a contiguous virtual address range for each cache tensor without backing the whole range with physical memory. The[vAttention paper](https://arxiv.org/abs/2405.04437)maps physical pages into that range on demand through CUDA virtual memory APIs. The attention kernel still indexes a conventional contiguous tensor. PagedAttention uses software translation. vAttention uses hardware-assisted virtual translation. The physical cache is paged in both designs. Fanout's[PagedAttention guide](/blog/pagedattention-vllm-kv-cache)explains logical and physical blocks before comparing the two ownership models. ## PagedAttention puts paging in the kernel A block table gives the serving engine direct control. It can allocate small blocks, free them promptly, and let several sequences reference the same prompt blocks. [Hugging Face's PagedAttention guide](https://huggingface.co/docs/text-generation-inference/conceptual/paged_attention)notes that block sharing is useful for parallel sampling, where several outputs reuse one prompt. The cost is extra work on the attention path. Each kernel needs code for block-table lookup, non-contiguous addresses, boundary conditions, and supported block sizes. The original paper measured 20 to 26 percent higher attention-kernel latency than its FasterTransformer baseline. It still won end to end because much better cache utilization admitted larger batches. A slower kernel can still produce a faster server when the alternative strands enough HBM to reduce concurrency. Block size controls the trade. Larger blocks improve parallel access and reduce table work, but the unused tail of each request grows. Smaller blocks reduce tail waste while increasing metadata and lookup work. Fanout's[continuous batching guide](/blog/continuous-batching-llm-inference)shows why admitting one more sequence can matter more than a small per-step kernel difference. ## vAttention keeps the virtual tensor contiguous vAttention separates virtual reservation from physical commitment. The engine presents a large PyTorch tensor address range, then attaches GPU pages only for tokens that exist. An ordinary contiguous-cache attention kernel can use the tensor without a block table. That lets a serving system reuse new FlashAttention or FlashInfer kernels without first building a paged variant. The authors report up to 1.23 times higher serving throughput than PagedAttention-based FlashAttention and FlashInfer kernels in their evaluated configurations. That is not a universal speedup. The paper also found that the paging overhead shrinks at high batch sizes and long contexts, where the attention work dominates address translation. The portability claim is broader than the benchmark. A kernel optimized for contiguous K and V can enter the serving stack with fewer layout-specific changes. This also keeps address translation out of the serving framework. CUDA page tables perform the virtual-to-physical mapping that the PagedAttention block table represents in user space. Fanout's[FlashAttention guide](/blog/why-is-flash-attention-faster)covers the kernel-level memory traffic that remains after cache allocation is solved. ## Page granularity sets the memory bill CUDA's standard low-level virtual memory path allocates physical memory at a minimum granularity of 2 MB on the GPUs evaluated by the vAttention authors. A 2 MB page can cover many decode steps, but the final page for each mapping may be mostly empty. The waste depends on model shape, tensor parallelism, dtype, and active request count. The vAttention team modified the open-source NVIDIA UVM driver to support 64 KB, 128 KB, and 256 KB pages. Its[reference implementation](https://github.com/microsoft/vattention)can also run with standard 2 MB pages. Smaller pages reduce unused tails. In the paper's model set, 64 KB pages had a theoretical maximum waste of 4 to 15 MB per request. At 256 KB, the range rose to 16 to 60 MB. Those figures include the model's multiple cache mappings, not one isolated page. They should not be transferred to another architecture without recalculating its cache layout. PagedAttention can use token blocks chosen by the engine rather than the CUDA allocation granularity. That flexibility is useful when memory pressure is severe or requests finish at highly variable lengths. vAttention's smaller-page result also has an operational cost: it depends on a modified driver. Standard 2 MB pages avoid that change but can waste more memory. ## Allocation latency must stay off the decode path Mapping a CUDA page requires a trip through the driver. If the engine waits synchronously, an allocation can interrupt a token step. In one Llama 3 8B experiment, the vAttention paper saw decode iterations rise from 25 ms to 41 ms when a new 2 MB page was mapped synchronously. About 4 ms of allocation latency applied per request. With allocation overlapped against the previous model iteration, the visible spike disappeared in that experiment. The system also allocates eagerly, reclaims pages later, and reuses pages from completed requests. These policies turn driver calls into background work instead of blocking the current token. The paper measured up to 15 percent synchronous prefill overhead with 64 KB pages. Larger 256 KB and 2 MB pages reduced that overhead to as little as 3 percent before the overlap optimizations. A contiguous kernel is not enough. The memory manager must stay ahead of cache growth at the workload's allocation rate. For decode, the paper measured at most 600 MB per second of required allocation and at least 7.6 GB per second of allocator capacity with its 64 KB configuration. ## Prefix sharing changes the comparison PagedAttention shares a prompt by pointing several logical block tables at the same physical blocks. Copy-on-write can give a branch a private block only when its continuation diverges. The original paper measured 37.6 to 55.2 percent memory saving from sharing on one beam-search setup, and 44.3 to 66.3 percent on its ShareGPT beam-search experiment. Those gains depend on branching. A simple chat with one continuation does not create the same opportunity. vAttention can represent sharing through virtual mappings, but the mapping and copy-on-write policy still need serving-system support. Contiguous virtual addresses do not create prefix reuse by themselves. Ask what the workload does. Parallel sampling, beam search, and branching agents make fine-grained shared blocks more valuable. Independent chats put more weight on kernel speed and portability. Prefix reuse also interacts with eviction. A shared block may outlive one request because another still references it, so cache accounting must follow physical ownership rather than request count. ## Choose the memory owner, then benchmark PagedAttention is the safer default when the deployed engine already has mature paged kernels, fine block sizing, prefix-cache policies, and production observability. vAttention is attractive when paged-kernel maintenance blocks a faster contiguous kernel, and the deployment can support CUDA VMM plus an allocator that hides mapping latency. Start by measuring cache fragmentation and admitted batch size. If fragmentation is not reducing concurrency, changing the memory manager may only move complexity. Then compare identical traffic. Record time to first token, inter-token latency, output throughput, HBM used, cache hit rate, and page or block allocation stalls. Test short and long contexts separately. The vAttention paper found that paged-kernel overhead can fade as attention work grows, so one context length cannot select the design. Finally, test prefix-heavy traffic if the product branches prompts. PagedAttention's block sharing can repay kernel overhead in a way that a linear decode benchmark never shows. Use CUDA translation when contiguous-kernel reuse is worth the page and driver constraints. Use software blocks when fine allocation and mature sharing matter more. --- This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included. Browse the public content map: https://fanout.sh/sitemap.md