Paged KV cache block size: how to choose

Paged KV cache block size is a trade between the unused tail of every live sequence and the kernel, scheduler, and cache metadata work created by more blocks.

There is no portable best paged KV cache block size. Sixteen tokens is common, but current engines expose different defaults and constraints.

The right number is the smallest supported block that improves usable capacity without hurting latency.

Start with the engine default. Calculate the maximum tail waste, test prefix reuse, then benchmark only the block sizes that the selected attention backend supports.

What paged KV cache block size controls

PagedAttention splits each sequence's KV cache into equal physical blocks. A block table maps logical token ranges to blocks that can sit anywhere in GPU memory.

If the block size is B tokens, a sequence of length L needs ceiling of L divided by B blocks. Its allocation is that block count multiplied by B token slots.

Only the final block can be partly empty. The wasted slots are B minus L modulo B, except when L divides B and waste is zero.

For a 47-token sequence, B equals 16 needs three blocks and allocates 48 slots. One slot is empty. B equals 32 needs two blocks, allocates 64 slots, and leaves 17 empty.

That is internal fragmentation. Paging removes the need for one large contiguous reservation, but it does not make the last partially filled block disappear.

Fanout's PagedAttention explainer covers the block table and copy-on-write model before this tuning decision.

Calculate tail waste before benchmarking

For a varied workload, sequence endings often land at many positions within a block. Under a roughly uniform remainder, expected tail waste is about B minus 1 divided by 2 token slots per live sequence.

At B equals 8, that expectation is 3.5 slots. At 16 it is 7.5. At 32 it is 15.5. This is a planning estimate, not a substitute for the actual length histogram.

Multiply expected wasted slots by the number of concurrent live sequences. Two hundred sequences at B equals 32 imply about 3,100 wasted token slots under the uniform assumption.

Then convert slots to bytes with the model's KV bytes per token. If one token uses 128 KiB, those tails occupy about 388 MiB.

The KV cache memory formula derives bytes per token from layers, KV heads, head dimension, precision, and tensor parallel size.

Use the real histogram for the final number. For each observed total sequence length, round up to the candidate block size, subtract the length, and sum the empty slots across active requests.

Do this on concurrency-weighted snapshots. A long request that stays active for ten minutes contributes more memory pressure than a short request that finishes in one second.

Smaller blocks are not free

Halving B roughly doubles the number of block references needed for the same token count. A 32K sequence needs 2,048 blocks at B equals 16 and 4,096 at B equals 8.

The attention kernel must gather K and V through that layout. More blocks mean a longer block table and more block boundaries for scheduler and cache bookkeeping.

The current vLLM PagedAttention design makes BLOCK_SIZE a compile-time kernel parameter and lays K and V out around it.

That detail explains why an arbitrary integer is not a harmless allocator setting. The chosen kernel must have a compatible specialization and memory layout.

A Google result led by FlashAttention issue 1579 shows the backend constraint directly. Its ROCm paged path required a block size divisible by 128.

That does not mean every engine should use 128. It means the feasible set comes from the exact backend, GPU, dtype, and engine version before workload tuning begins.

Current vLLM accepts a block size or lets its configuration resolve a default. Its documentation no longer promises one universal value across every supported model path.

Prefix reuse changes the cost

Prefix caching reuses KV state only at boundaries the engine can identify. Coarser physical blocks have historically made those boundaries farther apart.

The vLLM prefix caching design hashes the tokens in each block with the parent block hash. Its baseline design caches full blocks.

With B equals 32, a repeated 47-token prefix has one reusable full block and 15 tokens in a partial tail under that baseline. B equals 16 exposes two full blocks and leaves 15 tokens partial.

Larger blocks can therefore reduce reuse when prompts diverge near chat turns, tool-call boundaries, retrieved documents, or tenant-specific suffixes.

The engine is evolving here. Current vLLM also exposes a prefix-match unit that can be finer than a physical block for supported hybrid configurations.

Keep physical allocation granularity and prefix hash granularity separate when the version supports it. Changing block size to fix cache hits may be unnecessary if a finer match unit is available.

Measure reused tokens, prefill tokens recomputed, cache hit rate, and time to first token. A nominal hit rate can stay flat while the number of useful tokens per hit changes.

Engine defaults answer different constraints

One reason page-one advice conflicts is that engines do not share one block contract.

The TensorRT-LLM cache reuse documentation describes a 128-token default in that release and accepts power-of-two values such as 32.

It also states the trade plainly: larger blocks can improve kernel efficiency but reduce the chance that a reusable prefix fills complete blocks.

vLLM has commonly used much smaller attention blocks. Hybrid attention and Mamba models can require larger aligned physical blocks, then split those blocks for a kernel that consumes a smaller unit.

An engine default is therefore a compatibility decision across kernels and model families. It is not a measurement of your output-length distribution.

Do not copy a value from TensorRT-LLM into vLLM, or from one vLLM backend into another, without checking accepted values and the selected kernel.

Keep the default when the engine does not expose a supported alternative. A configuration flag that parses but triggers a fallback is a different experiment from changing one block size.

Benchmark SLO-qualified capacity

Block-size tuning matters only when it changes how many useful requests the server completes under its latency objective.

Use a trace that preserves prompt lengths, output lengths, arrival bursts, shared prefixes, and cancellations. Synthetic fixed-length requests erase the tail and reuse effects under test.

For every supported candidate, hold model, dtype, backend, GPU memory budget, scheduler limits, and request trace constant. Warm the server before collecting results.

Record available KV blocks after startup, live block occupancy, tail slots, prefix reuse, preemptions, time to first token, inter-token latency, and completed requests.

Fanout's continuous batching guide explains why batch occupancy changes while requests arrive and finish.

Report goodput rather than peak tokens per second. A larger block may speed the kernel slightly yet admit fewer concurrent requests because tail waste consumes the last usable cache capacity.

A smaller block may recover memory but lose the same requests to worse inter-token latency. Neither change is a win unless more work stays within the service objective.

Repeat the run. Near a capacity boundary, one burst can change preemption order and make a single trial look decisive when it is not.

When block size will not fix the problem

If weight memory leaves almost no room for the KV pool, quantization or another GPU will move capacity more than shaving a few tail slots.

If most requests are long, B equals 16 versus 32 changes at most 16 extra tail slots per sequence. Against tens of thousands of used slots, the percentage is small.

If prefix hit rate is low because prompts differ before the first full block, smaller blocks can help only within that first boundary. Canonical prompt construction may matter more.

If the backend requires a coarse page, an unsupported smaller value is not a tuning option. Changing attention backend may also change latency enough to dominate the memory gain.

If the service is scheduler-bound or compute-bound, block metadata may not be the limiting resource at all. Profile before assigning every capacity miss to the cache allocator.

Block size tunes granularity. It cannot repair an incorrect memory budget, a weak batching policy, or a workload that never reuses prefixes.

A decision rule for paged KV cache block size

Leave the engine default in place until measurements show one of two problems: tail waste blocks useful concurrency, or prefix boundaries discard enough reuse to hurt prefill latency.

List the block sizes supported by the exact model, backend, GPU, and dtype. Eliminate any candidate that changes kernels or falls back unless that broader change is intentional.

Calculate tail bytes from the live sequence-length histogram. If the savings cannot admit another request at the target context length, the smaller block has little capacity value.

Then replay production-shaped traffic and choose the candidate with the most SLO-qualified completions, provided quality and stability stay unchanged.

Paged KV cache block size is not a magic 8, 16, or 128. It is an allocation and kernel boundary whose value depends on where real sequences end and which cache states the engine can reuse.