Prefix match unit vs KV cache block size

Prefix match unit vs KV cache block size is a comparison between lookup granularity and physical storage granularity. Changing one does not automatically change the other.

In current vLLM, the split matters mainly for hybrid models whose cache groups need different physical block sizes. The scheduler may align work at 1,024 tokens while prefix hashes exist every 32 tokens.

Use the physical block size to reason about allocation and kernels. Use the prefix match unit to reason about where a reusable prompt may end. Then check the scheduler boundary that coordinates every cache group.

Prefix match unit vs KV cache block size

A KV cache block holds model state for a fixed number of token positions. Its size shapes page allocation, block tables, tail waste, and the memory layout expected by an attention kernel.

A prefix match unit says how many tokens feed each link in the prefix hash chain. A smaller unit creates more candidate boundaries where a later request can reuse earlier computation.

The current vLLM cache configuration says the option controls matching granularity, not how often states are stored.

That sentence prevents the most common mistake. Setting prefix_match_unit to 32 does not turn a 1,024-token physical state block into 32 separate allocations.

Fanout's paged KV cache block-size guide covers the allocation side: tail waste, kernel support, block-table length, and capacity.

The match unit belongs to the content-addressed lookup side. It changes the number of hashes and the amount of a repeated prefix that can be recognized.

The scheduler carries a third block size

Current vLLM resolves a scheduler block size and a hash block size.

The resolver source makes their jobs explicit.

For multiple cache groups, the scheduler size is the least common multiple of the groups' effective block sizes. That boundary is divisible by every group size, so one scheduling decision can keep the groups aligned.

The hash size is the configured prefix match unit, or the greatest common divisor of prefix-cacheable group sizes when no override is set.

Take two prefix-cacheable groups with effective block sizes 1,024 and 32. Their least common multiple is 1,024, while their greatest common divisor is 32.

The engine can therefore schedule on a 1,024-token invariant and build hashes every 32 tokens. Neither number replaces the two physical group sizes.

For a single cache group, the resolver returns that group's block size for both values. A finer prefix match unit is not a general way to subdivide ordinary attention-only pages.

This distinction is why the setting appears in hybrid-model work rather than basic PagedAttention tuning.

A 1,500-token prefix shows the gain

Suppose request A and request B share their first 1,500 tokens and diverge at token 1,501. The hybrid scheduler size is 1,024 tokens.

If cache hits must land on scheduler boundaries, only the first 1,024 tokens qualify. Request B recomputes the remaining 476 shared tokens before reaching its unique suffix.

With a 32-token match unit and compatible partial-hit managers, the largest full hash boundary is floor of 1,500 divided by 32, times 32. That is 46 times 32, or 1,472 tokens.

Request B now recomputes 28 shared tokens. The finer hash boundary saves 448 token positions of repeated prefill in this example.

The original vLLM partial-cache-hit RFC uses a smaller worked sequence to show the same mechanism.

It hashes every two tokens inside a six-token physical block. A later request can match an eight-token prefix, then use copy-on-write before appending its divergent suffix.

The benefit depends on where real prompts diverge. If shared prefixes already end near 1,024-token boundaries, a 32-token hash unit adds work without recovering much more prefill.

Finer hashes do not mean finer physical pages

At 128K tokens, a 1,024-token hash unit produces 128 hash positions. A 32-token unit produces 4,096, which is 32 times as many lookups and hash-chain entries.

That count is not the KV storage count. The cache groups still allocate their own blocks, and the hybrid coordinator maps a recognized prefix onto compatible stored state.

The hybrid KV cache manager design explains why groups exist.

Full attention, sliding-window attention, and Mamba retain different state. The coordinator must find a prefix length that every participating group can safely reuse.

Full attention accepts a prefix only while all earlier blocks remain cached. A sliding-window group can reason from the recent window. A Mamba group needs a recurrent state valid at the chosen boundary.

The final cache hit is therefore an intersection, not the longest match reported by whichever group has the smallest unit.

Fanout's PagedAttention explainer is the useful prerequisite if logical-to-physical block mapping is still the confusing part.

The override has strict validity rules

Every prefix-cacheable group block size must be divisible by the chosen prefix match unit. Group sizes 1,024 and 32 accept a unit of 32, 16, or 8, but not 48.

The resolver also checks state-compression alignments. A hash boundary that cuts through one indivisible recurrent state cannot become a safe resume point.

Fine partial hits currently require a compatible Mamba group in align mode. If Mamba uses another cache mode, vLLM falls back to scheduler-sized hash behavior.

Every participating cache manager must also support fine-grained hash lookup. The current coordinator disables the feature if one incompatible manager would need a smaller-than-block lookup.

Prefix caching or a KV connector must be active for the finer hashes to have a consumer. Without either, the resolver keeps hash size equal to scheduler size.

Decode context parallelism changes effective attention block sizes before the least common multiple and greatest common divisor are calculated.

These are versioned implementation constraints. Check the exact vLLM revision and startup logs instead of assuming that a parsed flag produced partial hits.

Measure reusable tokens, not hit count

A cache-hit counter can give one 1,024-token hit and one 1,472-token hit the same weight. The second avoids 448 more prefill tokens, which is the result users feel.

Record matched tokens, recomputed prompt tokens, time to first token, and CPU time spent creating and looking up hashes.

Break the data down by prompt family. Stable system prompts, tool schemas, chat turns, and retrieved documents tend to end at different token positions.

Also record eviction. A finer lookup boundary cannot recover a prefix whose required physical state has already left the cache.

Replay the same request order for each candidate. Prefix caching is stateful, so changing arrival order or cache warmth can overwhelm the effect of the match unit.

Use production tokenization. A boundary measured in characters or JSON fields does not tell you where the tokenizer places the 32-token cut.

Fanout's chunked prefill guide explains a separate scheduler mechanism. Prefill chunk size controls how new work is interleaved, not where an old prefix hash can match.

Tune from the default split

First inspect the resolved group sizes, scheduler block size, and hash block size at server startup. Do not tune a value that the engine already resolved to the desired greatest common divisor.

Build a histogram of shared-prefix endings modulo the scheduler size. Estimate how many extra tokens each smaller valid match unit would recover.

Eliminate units that violate group divisibility, recurrent-state alignment, or manager support. The theoretical set of divisors is larger than the runnable set.

Replay a representative trace and compare useful matched tokens with hash overhead, time to first token, preemptions, and SLO-qualified throughput.

Keep the physical KV cache block sizes unchanged during this test. If both storage and matching change, the result cannot identify which lever moved capacity or latency.

Prefix match unit vs KV cache block size becomes manageable once the three boundaries have names. Physical blocks store state, the scheduler aligns groups, and hash units expose reusable prefix endings.