Prefix cache retention with speculative decoding

Prefix cache retention with speculative decoding can fail in a pattern that looks impossible: the cold request misses, the first identical repeat also misses, and only the second repeat hits.

That miss-miss-hit trace is not ordinary eviction. On a hybrid model, speculation can move the reusable boundary while sparse retention keeps a different recurrent-state checkpoint.

The full-attention cache may recognize the prefix at the shifted boundary. If one Mamba, GDN, or sliding-window group has no state there, the hybrid hit falls back to zero.

Why prefix cache retention and speculative decoding collide

A hybrid cache hit is bounded by the least reusable cache group. Full attention may retain token-growing KV blocks, while a recurrent group retains selected state snapshots.

Fanout's hybrid cache memory guide explains why these groups have different storage shapes. A valid hit still needs a boundary every required group can execute from.

Speculative methods add another boundary rule. EAGLE-family and MTP paths can reserve or drop a block so the target and draft state remain consistent during verification.

That adjustment can move the usable boundary behind the prompt's final aligned checkpoint. The scheduler and lookup may agree on the adjusted position while the retention mask kept only the unadjusted position.

vLLM's engine argument reference defines the retention interval separately from the prefix match unit.

The match unit controls where a hit may land. The retention interval controls which sliding-window and Mamba checkpoints remain available for a later request.

Those controls can be individually valid and jointly misaligned. A fine lookup grid cannot recover a recurrent state that the producing request never registered.

Read miss-miss-hit as a boundary trace

A vLLM issue reproducing the first-repeat miss used an approximately 12,000-token prompt on a hybrid model with three Mamba cache groups and one full-attention group.

With MTP enabled, the cold request missed, the first repeat re-prefilled, and the second repeat hit. Without speculative decoding, the first repeat hit immediately.

The reported prompt had 11,910 tokens. Its scheduler and cache block size were 1,600 tokens, so the ordinary final full boundary was 11,200 tokens.

MTP used EAGLE-family cache semantics and backed the reusable boundary off by one 1,600-token unit. The boundary that every group needed was therefore 9,600 tokens.

During request one, the Mamba state at 9,600 existed but the sparse retention mask rejected it. The Mamba groups retained the 11,200 replay boundary instead.

Full attention registered 9,600. On request two, full attention found that boundary but the Mamba groups did not. Reconciliation returned a zero-token hybrid hit.

The failed lookup also exposed 9,600 as an uncached common prefix. Request two then registered the missing Mamba state at that shared junction, so request three hit 9,600.

The trace is causal: production skipped the adjusted boundary, the first repeat learned it, and the second repeat reused it. A simple LRU eviction would not predict that exact progression.

Calculate the shifted boundary before tuning

Start with four numbers: prompt length, scheduler block size, final aligned boundary, and the speculative adjustment.

For the reported case, floor of 11,910 divided by 1,600 is 7. Multiplying by 1,600 gives the 11,200-token aligned boundary.

The EAGLE-family adjustment removes one block unit. Subtracting 1,600 gives the 9,600-token reusable boundary.

Now inspect each required group at 9,600. Record whether its state was materialized, whether the retention mask marked it reachable, and whether a cache hash was registered.

The distinction between materialized and retained matters. If a state existed during request one but no cache key points to it later, allocating more KV memory does not repair the lookup contract.

Fanout's prefix match unit comparison covers the difference between hash boundaries and physical cache blocks.

Add retention to that worksheet. For every candidate boundary, note whether each group can look it up and whether the scheduler can resume computation there.

Do not average hits across groups. A 9,600-token full-attention hit plus three zero-token Mamba hits does not produce a partial 2,400-token hybrid hit. The executable common boundary is zero.

Separate retention bugs from reporting bugs

Older vLLM combinations could run prefix caching and speculative decoding together without exposing complete prefix-hit metrics. A missing chart therefore did not always mean a missing cache hit.

The current diagnosis needs request-level evidence: cache query deltas, cache hit deltas, TTFT, reused token counts, and the per-group boundary selected by lookup.

Use three serialized identical requests. Let each request finish before sending the next, so overlap and an in-flight producer cannot explain the first repeat's miss.

A reporting problem has stable latency and reuse despite absent counters. A retention mismatch shows fresh prefill work on the first repeat and a later transition to a real hit.

The affected vLLM reproduction measured 24.9 seconds cold, 2.83 seconds for the full re-prefill, and 0.61 seconds for the later hit on an unmodified build.

The control without speculation produced a miss-hit pattern instead. That paired test is stronger than comparing one configuration to a theoretical expectation.

A separate DFlash cache investigation also corrected an initial attribution after the draft-group annotation fix did not change the slow prefix child.

Its measured improvement came from retention and chunk granularity. That is a useful warning: a plausible metadata bug can coexist with a different performance cause.

Test the interval workaround without calling it free

The first-repeat report set the retention interval to the 1,600-token block size. Cold TTFT was 2.52 seconds, and four repeats each reached about 0.55 seconds.

That result shows that denser retained checkpoints can cover the EAGLE-adjusted boundary. It does not show that the setting is capacity-neutral under long-context pressure.

Retaining more recurrent checkpoints consumes cache blocks and changes eviction pressure. The correct test pairs hit recovery with resident-state occupancy and coexisting-conversation capacity.

An open per-group retention RFC reports the opposite failure mode: one EAGLE-style sliding-window draft group consumed 87 percent of the prefix cache.

The RFC proposes sparse retention for a hit-inert draft group while preserving dense checkpoints for target MLA or Mamba groups. Its measurements come from a fork, not verified upstream behavior.

That distinction rules out a universal rule such as "set retention to one block." The group that needs the shifted target boundary may differ from the draft group that makes dense retention expensive.

Sweep only valid interval multiples. For each setting, record first-repeat reuse, later-repeat reuse, cached block count by group, evictions, TTFT, and the longest two conversations that remain warm together.

Keep the prefix match unit fixed during this sweep. Otherwise a recovered hit could come from a different lookup grid rather than the retention change under test.

Diagnose extending children separately

An identical resend is the cleanest first test, but production traffic often shares a parent and adds a new child suffix.

The child may need a boundary before the new suffix rather than the prompt's final replay boundary. Prefill chunking can create or skip useful shared-prefix junctions along that path.

The DFlash investigation measured a 51,800-token child. Setting the retention interval to 832 tokens reduced TTFT from 35.15 seconds to 5.83 seconds and added 74,880 reused-token counts across the cache groups.

Reducing the maximum batched tokens to 1,664 cut the child to 1.47 seconds, at the cost of about 13 percent more time on the cold parent prefill.

The retained checkpoint and the chunk boundary were both part of the outcome. An exact resend alone would not reveal that child-prefix tradeoff.

Test four shapes: exact resend, one-token extension, divergence just before an aligned boundary, and divergence just after it.

For each shape, log the full-attention match, recurrent-state match, reconciled hit, uncached common prefix, and fresh prefill tokens.

That matrix separates a missing checkpoint from a coarse match unit, a chunk split, or a draft-group identification problem.

Use a stop rule for production changes

A retention change is ready only when the first valid repeat hits the intended boundary and outputs still match the control.

It also needs a capacity check. Measure cache occupancy, evictions, and warm-conversation survival at the longest supported contexts instead of relying on one request on an empty worker.

Do not treat a later hit as proof that the first miss is harmless. The first customer retry still pays the re-prefill, and reactive junction learning may disappear after eviction or routing to another worker.

Do not generalize fork measurements to upstream vLLM. Record the exact commit, model cache layout, speculative method, block size, match unit, retention setting, and batch-token limit.

The practical diagnosis is narrow. Compute the speculative boundary, verify every required group retained executable state there, and reproduce the request sequence that exposes the gap.

When miss-miss-hit becomes a boundary table, the apparent cache mystery reduces to one missing recurrent-state checkpoint.