Prefix cache retention interval for hybrid models

A prefix cache retention interval for hybrid models decides which recurrent and sliding-window states remain reusable after a request finishes. It does not change every full-attention KV block into a larger page.

In current vLLM, zero keeps semantic checkpoints, a positive value adds periodic checkpoints, and no value keeps eligible checkpoints densely.

The choice trades cache occupancy against the amount of a repeated prompt that must be recomputed.

Start with the workload's reusable boundaries. Long conversations and branch-heavy agents need different policies even when they run the same model.

Prefix cache retention interval for hybrid models

A hybrid model may combine full attention with Mamba, another linear-attention state, or sliding-window attention. Each cache group needs different state to resume at a prefix boundary.

Full attention retains keys and values for the entire reusable prefix. A recurrent layer needs a valid state at the chosen boundary. A sliding-window layer needs the recent window leading into that boundary.

The current vLLM CLI reference applies prefix cache retention only to sliding-window and Mamba groups. Full-attention and chunked-local groups keep their normal dense behavior.

Raising the interval does not make full-attention prefix hashes coarser. It makes the sparse-state groups retain fewer periodic resume points.

Fanout's prefix match unit guide separates hash granularity, scheduler alignment, and physical block size. Retention is a fourth question: which valid states survive for reuse.

Three settings produce three retention policies

The default value is zero in the current CLI. Zero keeps semantic checkpoints such as the latest replay boundary and shared-prefix junctions, but skips ordinary periodic checkpoints.

A positive value keeps those semantic checkpoints and adds periodic states at that token interval. The value must be a multiple of the scheduler block size.

An unset value, represented as None in the implementation, retains eligible checkpoints densely. This gives later requests more nearby resume points and consumes the most cache-block IDs for sparse groups.

The official Kimi K3 serving note frames the same choice for agentic traffic.

It recommends prompt-end states for multi-turn conversations and periodic states for useful prefixes that may occur far from a prompt boundary.

These modes do not express seconds. The interval is measured in tokens, so a 14,336-token setting spans different wall-clock times under different prefill rates.

Why one retained boundary can cost many blocks

A full-attention group can associate one block at a scheduler boundary with one prefix-cache entry. Sparse groups may need more state to make that same boundary resumable.

For sliding-window attention, the manager needs enough contiguous blocks to cover the preceding window. With a window W and block size B, the count is ceiling of (W minus 1) divided by B, plus 1.

Take a 2,048-token sliding window and 64-token blocks. The count is ceiling of 2,047 divided by 64, plus 1, which equals 33 blocks at each retained boundary.

Now add one full-attention group and four aligned Mamba groups. One boundary can hold 1 full-attention block, 4 recurrent states, and 33 sliding-window blocks, or 38 block IDs in all.

This geometry comes from a recent vLLM retention RFC. Its deployment numbers are evidence from a fork, not an upstream performance guarantee.

The arithmetic still exposes the mechanism. Counting one boundary as one cache entry can understate its occupancy by more than an order of magnitude.

Work the interval against an 80K-token prompt

Suppose the scheduler block size is 3,584 tokens and a repeated prompt is 80,000 tokens long. It spans 22 full scheduler segments with a short tail.

Dense retention can keep an eligible sparse-group checkpoint at every one of those 22 segment boundaries. The next branch can resume close to its shared-prefix ending.

Set the positive interval to 14,336 tokens, or four scheduler blocks. Only every fourth ordinary boundary becomes a periodic sparse-state checkpoint, in addition to semantic checkpoints.

The 22 ordinary boundaries now yield about five periodic checkpoints instead of 22. The exact total also includes prompt ends and learned shared-prefix junctions.

The price is recomputation. A branch that ends just before the next periodic boundary may need to rebuild almost 14,336 tokens of sparse-group state from the previous retained checkpoint.

Full-attention reuse can still extend farther. The coordinator must reconcile what every required group can safely resume, so the sparse state may determine the usable prefix.

For one prompt family, compute retained block IDs per boundary times expected live boundaries. Compare that number with recomputed tokens per request, not only cache-hit percentage.

Work the checkpoint and replay arithmetic

Let T be a processed sequence length, I the positive retention interval, and S the bytes in one retained hybrid-state checkpoint for the request.

Ignoring prompt-end additions and alignment, periodic checkpoint count is about T divided by I. Their state cost is about S times T divided by I.

For T = 196,608 tokens and I = 32,768, the sequence crosses six periodic intervals. A prompt-end checkpoint may add one more retained state if it does not coincide with an interval boundary.

If future shared-prefix endpoints are spread uniformly inside an interval, the average gap to the previous periodic checkpoint is about I divided by two. The worst gap is just under I tokens.

That average is an assumption about workload shape, not a vLLM guarantee. Agent traces cluster at prompt ends, tool boundaries, and shared instructions rather than at uniformly random token positions.

The actual saved boundary is also aligned to the cache manager's valid block or state boundary. Treat I as a retention cadence, then inspect the resolved cached-token count in the running engine.

Fanout's paged KV cache block-size guide covers the physical allocation unit that sits under this arithmetic.

A fine hash cannot recreate a missing state

Suppose the prefix matcher identifies a shared prefix ending at token 50,176, while the previous retained hybrid checkpoint is at token 32,768.

Full-attention KV may exist through the longer match. The recurrent group still cannot resume from a state that was never retained at token 50,176.

The safe reusable boundary is constrained by the least reusable cache group. The engine must fall back to a retained state and replay the missing hybrid suffix, subject to its alignment rules.

Reducing a 32-token prefix match unit to 16 tokens produces more hash candidates. It does not create twice as many Mamba snapshots or sliding-window tails.

Likewise, increasing I from 32,768 to 65,536 thins hybrid checkpoints. It does not make hash lookup coarser and does not double the physical KV page size.

This distinction explains an odd metric pattern: a router can identify a long shared token prefix while the serving engine reports fewer reusable computed tokens.

Semantic checkpoints change the workload answer

Zero does not mean no sparse-state caching. It keeps boundaries that have an immediate semantic reason to be reused.

The latest replay boundary is useful for a normal next turn because the conversation replays the prior prompt before appending new input. A prompt-end checkpoint can avoid rebuilding the entire recurrent history.

Shared-prefix junctions cover another pattern. When the system sees a prefix reused, it can retain the state where requests diverge instead of predicting every useful branch in advance.

This suits long chats with one continuation at a time. Dense periodic retention would preserve many internal positions that no later request ever selects.

Branch-heavy agent workloads differ. A repository snapshot, tool schema, or long instruction prefix may fork before the most recent prompt end. A positive interval limits the worst recomputed suffix for those branches.

Measure boundary demand from real tokenized prompts. Character counts and message counts do not identify the scheduler-aligned state that the engine can reuse.

EAGLE and MTP can move the reusable boundary

Speculative decoding adds another cache group and can adjust the final reusable block. That boundary must agree with the state retained by the target model's Mamba or sliding-window manager.

A current vLLM issue reports a hybrid Mamba model whose first repeated prompt missed with MTP enabled. The second repeat then hit the retained adjusted boundary.

In that reproduction, a positive retention interval equal to the 1,600-token scheduler block restored an immediate hit. The report measured the workaround on named nightly builds and hardware.

Treat that result as a diagnostic pattern, not a general setting. A newer build may change boundary handling, and a different speculative method may retain another state shape.

Run a cold request followed by at least two serialized repeats. Record matched tokens and time to first token for each repeat with speculative decoding both on and off.

One warm repeat is not enough. A miss, miss, hit sequence points to retention timing or boundary alignment that a single hit-rate aggregate can hide.

A global interval can penalize the wrong group

Current upstream code passes one retention interval from the coordinator to every eligible sparse group. Each manager interprets that scalar using its own block size and state requirements.

That is simple, but hybrid groups can have very different occupancy. A small EAGLE sliding-window group may retain dozens of blocks per boundary while a target-model Mamba group retains one state.

Increasing one global interval reduces both groups' periodic checkpoints. It can recover capacity consumed by the expensive group while making a useful target-model prefix coarser too.

The recent per-group RFC proposes resolving separate intervals and exempting a hit-inert draft group from the final hit minimum. The proposal was still an open design discussion when this draft was written.

Do not configure behavior that exists only in the RFC. Use the current coordinator source to verify what the deployed revision supports.

Version-pin the test result. Retention defaults, validation rules, and group semantics have changed quickly enough that a copied environment variable is not a durable runbook.

Choose the interval from measured cost

Start with zero for multi-turn traffic whose next request usually extends the last prompt. Confirm that prompt-end states survive long enough to serve the next turn.

Try a positive interval for branchy agents and repeated long prefixes whose useful divergence points are not predictable. Test multiples of the resolved scheduler block size.

Use dense retention only when nearby resume points save enough prefill to justify their cache occupancy. Long exact repeats may need only the prompt end, so dense can waste space even there.

For each setting, record retained block IDs by group, useful matched tokens, recomputed prompt tokens, evictions, time to first token, and prefix-cache lookup time.

Replay the same arrival order. Prefix caching is stateful, and a different sequence can change which blocks the shared LRU evicts.

The PagedAttention guide explains the physical block pool behind those evictions. Retention decides which sparse states compete inside that pool.

A workable prefix cache retention interval for hybrid models is the coarsest policy that preserves the branches users actually repeat. The correct unit is saved prefill per retained block, checked against first-token latency.