--- title: "Hybrid prefix cache lost tokens metric" description: "Use sparse-retention misses with prefix queries and hits to find shared tokens that a hybrid cache matched but could not safely reuse." canonical_url: "https://fanout.sh/blog/hybrid-prefix-cache-lost-tokens-metric" md_url: "https://fanout.sh/blog/hybrid-prefix-cache-lost-tokens-metric.md" last_updated: "2026-10-08" access: "public" --- # Hybrid prefix cache lost tokens metric Use sparse-retention misses with prefix queries and hits to find shared tokens that a hybrid cache matched but could not safely reuse. - Author: Suraj Gaud - Published: 2026-10-08 - Track: Inference engineering - Access: Public - Tags: prefix caching, hybrid models, vLLM, Mamba, sliding window attention, cache metrics, TTFT A hybrid prefix cache lost tokens metric answers a question that ordinary hit rate cannot: did the requests share nothing, or did one cache group find shared work that another group vetoed? Ordinary hit rate merges those two cases on models mixing full attention with Mamba, GDN, or sliding-window layers. Attention may match a prefix while sparse state has no checkpoint at the same boundary. The safe hybrid hit is then shorter than the matched attention prefix, sometimes zero. One open vLLM change proposes counting the shared tokens lost specifically to that sparse-retention gap. ## What the hybrid prefix cache lost tokens metric counts Hybrid cache lookup is an intersection. Every active cache group must supply state from one boundary where the scheduler can resume the model correctly. Full attention stores token-growing key and value blocks. A recurrent group stores selected state snapshots. A sliding-window group needs the recent window that makes a candidate boundary valid. Fanout's[hybrid cache memory guide](/blog/hybrid-kv-cache-vs-recurrent-state-memory)explains why these groups have different storage shapes. They also have different reuse rules. Suppose the full-attention group matches 1,088 shared tokens. If the Mamba group has no checkpoint at token 1,088, returning that attention hit would resume the recurrent layers from invalid state. The coordinator must reduce the final hit. A plain hit counter reports only the reused result, so the request looks identical to one whose token IDs diverged at the start. The open[vLLM metrics change](https://github.com/vllm-project/vllm/pull/52527)proposes vllm:prefix_cache_sparse_retention_misses for the discarded token count. The proposal is observability only. It does not create a checkpoint, change cache lookup, or make an unsafe partial hit valid. ## Work through the miss, miss, hit sequence The PR includes a single-GPU check with three 2,176-token requests. Each request shares the first 1,088 tokens and has a different 1,088-token tail. The engine resolves 544-token blocks. The cold seed reports zero hits and zero sparse-retention misses because no earlier request has established a shared prefix. The second request finds the 1,088-token shared junction in the attention cache. The sparse group has no checkpoint there, so the final hit is zero and the proposed counter adds 1,088 tokens. While processing that request, the engine learns the shared junction and preserves suitable state. The third request then reuses 1,088 tokens and adds zero sparse-retention misses. Read the sequence as 0/0, 0/1,088, then 1,088/0 for hit tokens and lost tokens. The second request is the one an ordinary hit-rate chart mislabels as no sharing. The earlier[vLLM bug report](https://github.com/vllm-project/vllm/issues/45238)documents the underlying geometry on align-mode Mamba caches. Later comments narrow the original claim. Current admission and retention work recovers much of the catastrophic zero-hit case, but a geometry-dependent residual loss remains and is still hard to attribute. ## Read queries, hits, and lost tokens together The useful prefix-hit ratio is hit tokens divided by queried tokens. It measures the share of eligible query positions that the engine actually reused. The sparse-loss ratio is sparse-retention misses divided by queried tokens. It measures work that some cache group matched but the hybrid coordinator could not use because sparse state was absent. Add hit tokens and sparse-retention misses only as a diagnostic subtotal. It describes reuse found by the measured path, not a promise that every lost token would be free under a different setting. All three Prometheus counters are cumulative. Compare deltas over the same request window rather than dividing one fresh scrape by a value retained from an earlier deployment. In the PR's 32-request test, both geometries queried 70,400 tokens. The favorable case reused 38,016 tokens and recorded no sparse loss. The unfavorable case reused 17,952 tokens and recorded 8,448 sparse-retention misses. Its useful hit ratio was 25.5 percent, while sparse retention explained another 12 percent of queried tokens. Mean TTFT rose from 245.95 ms to 300.25 ms in that comparison, about 22 percent. Those numbers belong to the reported Qwen3.5-4B setup and are not a general vLLM forecast. The metric excludes reductions caused by dense-cache eviction and the intentional EAGLE replay drop. That attribution guard matters because both can shorten a hit without a missing sparse checkpoint. ## Separate four causes of a low hit rate First check token identity. Different templates, whitespace, tool schemas, or tokenization can eliminate the common prefix before any cache policy is involved. Second check routing and residency. A matching prefix on another worker or an evicted dense block produces a miss that sparse-retention metrics should not claim. Third check boundary granularity. A shared prefix shorter than the current hash unit may contain no legal lookup point even when all required state exists. Fanout's[prefix match unit guide](/blog/prefix-match-unit-vs-kv-cache-block-size)separates hash boundaries from physical pages and scheduler alignment. Fourth check sparse state. If attention finds a shared boundary but Mamba or sliding-window state is absent there, the sparse-retention counter should rise. A zero value does not prove caching is healthy. It rules out only the loss that the counter attributes. Dense eviction, cross-worker routing, disabled caching, and zero token overlap still need their own evidence. Likewise, a positive value is not permission to force the attention hit through. The missing recurrent or window state is a correctness boundary, not a conservative performance hint. ## Check retention and match geometry separately Current vLLM work describes a zero retention interval as keeping semantic checkpoints rather than every boundary state. That saves cache blocks but leaves some repeated prefixes without reusable sparse state. The same PR proposes a startup message for this configuration. It answers whether the server is expected to retain sparse checkpoints, while the runtime counter answers whether a request lost reuse. A smaller prefix match unit can expose more candidate boundaries. It does not make recurrent state appear at all of them. Periodic retention can preserve more states. It can also increase cache occupancy, evictions, and the number of warm conversations the server can hold. These knobs should move one at a time. First fix the retention policy and sweep valid match units. Then fix the match unit and sweep permitted retention intervals. Record resolved block sizes from startup. Hybrid layouts can select hundreds or thousands of tokens per physical block, so a copied flag value may be invalid for another model or quantization. The[SuffixReplay paper](https://arxiv.org/abs/2609.33477)explores a different design: reconstruct recurrent state by replaying a recent suffix instead of materializing a checkpoint at every reusable boundary. SGLang's[Unified Radix Cache design](https://www.lmsys.org/blog/2026-08-11-unified-radix-cache)takes another route, attaching component-specific validators to one shared token topology. Both designs enforce the same constraint. Full-attention prefix identity alone does not prove that every hybrid component can resume from that point. ## Build a three-request diagnostic Start with one cold request, then send two requests with the same known prefix and different suffixes. Serialize them so an in-flight producer cannot explain the first repeat's miss. Capture per-request deltas for prefix queries, prefix hits, and sparse-retention misses. Also record TTFT, fresh prefill tokens, resolved block size, match unit, and retention interval. If the second request records lost tokens and the third turns those tokens into hits, the cache is learning a junction after observing sharing. That is the miss, miss, hit signature. Repeat with a prefix ending just before and just after a legal boundary. The pair reveals whether small changes in suffix length move the retained checkpoint into request-unique tokens. Run one control without speculative decoding and another without cache pressure. These controls separate state geometry from an EAGLE adjustment or an eviction. Do not compare server-wide averages across different request mixes. A few long prefixes can dominate token counters while many short prompts dominate request counts. ## Use the metric as an attribution signal Alert on a sustained sparse-loss ratio beside rising TTFT, not on one cold request. A cold seed has no earlier shared state to lose and should not raise this counter. Break the ratio down by model revision, speculative method, block geometry, and prompt family. The failure depends on where shared prefixes end relative to valid sparse checkpoints. Treat a configuration change as successful only when lost tokens fall, useful hits rise, outputs match the control, and cache occupancy remains inside the service budget. The hybrid prefix cache lost tokens metric names the discarded work. Operators can then tune checkpoint geometry without weakening the rule that every cache group must resume from valid state. --- This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included. Browse the public content map: https://fanout.sh/sitemap.md