--- title: "Measure speculative index confirmation rate" description: "Define prefix-cache speculative index confirmation rate with mature cohorts, then separate event-pipeline health from actual prefill work avoided." canonical_url: "https://fanout.sh/blog/prefix-cache-speculative-index-confirmation-rate" md_url: "https://fanout.sh/blog/prefix-cache-speculative-index-confirmation-rate.md" last_updated: "2026-10-23" access: "public" --- # Measure speculative index confirmation rate Define prefix-cache speculative index confirmation rate with mature cohorts, then separate event-pipeline health from actual prefill work avoided. - Author: Suraj Gaud - Published: 2026-10-23 - Track: Inference engineering - Access: Public - Tags: prefix caching, llm-d, speculative indexing, KV cache, routing, observability, inference metrics Prefix cache speculative index confirmation rate asks how often a predicted cache placement becomes real before its TTL expires. Count unique predicted block-and-pod placements, then match them to later store events. That rate diagnoses the event path. It does not prove that speculation saved prefill work. A confirmed entry may never receive a second request, while a routed follower can still miss at the model server. Measure confirmation first, then connect speculative lookups to server-observed cached tokens and latency. ## Define prefix cache speculative index confirmation rate llm-d's[KV-cache indexer](https://llm-d.ai/docs/architecture/kv-management/kv-indexer)builds a view of cached blocks from model-server events. There is a short blind spot between routing a request and receiving the corresponding `BlockStored` event. Speculative indexing predicts that the selected pod will soon contain the request's newly computed blocks. The predicted entries live until a confirming store event arrives or the configured TTL expires. Current documentation describes a default TTL of two seconds. For one mature cohort, define P as unique speculative block-and-pod placements inserted. Define C as placements confirmed by a matching store event before expiry. The block confirmation rate is C divided by P. If 8,600 of 10,000 mature predictions confirm, the rate is 86 percent. Use unique placements rather than raw callbacks. Retries, duplicate events, and repeated references to the same predicted block must not inflate the numerator. ## Close the cohort before dividing A live denominator contains predictions that have not had time to confirm or expire. Counting them as failures makes the newest traffic look worse than older traffic. Assign each prediction an insertion time and expiry deadline. Close the cohort only after the deadline plus a small allowance for event ingestion and clock skew. Track three terminal states: - Confirmed before the deadline. - Expired without confirmation. - Invalidated because the pod or index generation disappeared. Keep pending predictions outside the mature denominator. Report them separately so a stalled event stream remains visible. Suppose a ten-minute window inserts 20,000 unique predictions. At query time, 16,400 are confirmed, 2,600 expired, and 1,000 remain pending. The mature denominator is 19,000, not 20,000. Confirmation rate is 16,400 divided by 19,000, or 86.3 percent. Expiry rate is 13.7 percent. If the pending count keeps growing after the TTL allowance, the measurement pipeline is unhealthy. The percentage alone would hide that backlog. ## Count blocks, routes, and tokens separately A request can predict many cache blocks. One early block may confirm while later blocks fail because the request ends, errors, or follows a different token path. Block confirmation rate measures index materialization. Request confirmation rate answers a different question: how many routed requests had all expected speculative placements confirmed. Token-weighted confirmation gives long prefixes appropriate weight. Sum the tokens represented by confirmed predicted blocks and divide by tokens represented by all mature predicted blocks. Report all three when the traffic mix varies. A 95 percent request rate can coexist with a much lower token-weighted rate if long requests lose their tails. Keep pod identity in the key. A matching block stored on pod B does not confirm a prediction that sent the request to pod A. Keep cache namespace in the key as well. Model, adapter, cache salt, multimodal hashes, and block geometry can separate otherwise identical token sequences. Fanout's[prefix match unit guide](/blog/prefix-match-unit-vs-kv-cache-block-size)explains why token matching and physical cache blocks may use different boundaries. ## Confirmation is not useful reuse A confirmation proves that the predicted placement agreed with a later cache event. It says nothing about whether another request consulted that entry. Add a second funnel. Count speculative lookups that influenced a route, then correlate the selected pod with the model server's actual cached-token result. One useful sequence is predictions inserted, predictions confirmed, speculative entries referenced, follower requests routed, and follower cached tokens observed. Each step can fall for a different reason. Low confirmation points to failed requests, event delay, identity mismatch, or a TTL that is too short. High confirmation with few references means the workload rarely repeats a prefix inside the useful window. The feature is accurate but does little work. Many references with few server hits suggest stale predictions, cache eviction, or a disagreement between router hashes and model-server hashes. The[precise routing guide](https://github.com/llm-d/llm-d/blob/main/guides/precise-prefix-cache-routing/README.md)requires the router and model server to agree on block size and model identity. That agreement is part of measurement correctness. A confirmation counter cannot repair a tokenizer or block-size mismatch. ## Work through an avoided-prefill example Assume the mature cohort contains 10,000 predicted placements covering 640,000 tokens. Of those, 8,600 placements covering 540,000 tokens confirm before expiry. The block confirmation rate is 86 percent. The token-weighted confirmation rate is 540,000 divided by 640,000, or 84.4 percent. Now suppose 2,000 follower requests consult speculative entries. Only 1,500 route to the predicted pod before the entries expire. The model servers report 360,000 cached prompt tokens for those followers. Do not call all 360,000 tokens a benefit of speculation without a baseline. Some followers may have reached the same pod through load scoring or ordinary confirmed index state. Compare with speculation disabled or use a controlled routing holdback. If the holdback group reuses 210,000 tokens at the same traffic volume, the measured incremental reuse is 150,000 tokens for that interval. Multiply incremental cached tokens by measured prefill time per token only when that cost is approximately linear for the workload. Otherwise use direct TTFT and accelerator-time measurements. Fanout's[prefix-cache retention guide](/blog/prefix-cache-retention-interval-hybrid-models)shows why a logical match may still be limited by the shortest-lived cache group. ## Use event health to explain a low rate The current llm-d[router metrics reference](https://github.com/llm-d/llm-d-router/blob/main/docs/metrics.md)exposes index admissions, evictions, lookups, active subscribers, reconnections, and skipped store events. Those metrics are disabled by default in shipped sample configurations. Enable them before treating an absent time series as a zero. Join confirmation rate with event rate and recent routing. Low events with low routing can mean the endpoint is idle. Low events with sustained routing can mean the event pipeline is broken. Subscriber reconnections, skipped stores, or replay gaps can narrow the cause. The llm-d[dynamic TTL proposal](https://github.com/llm-d/llm-d-kv-cache/issues/544)lists confirmation rate as planned health-monitor data rather than a complete benefit metric. Preserve that status in dashboards. If your release does not emit a native confirmation counter, label the derived metric with its collector and matching rules. Do not infer confirmation from index size. Evictions, duplicate blocks, and confirmed entries that outlive speculative entries all change the size independently. ## Choose a TTL from the latency distribution A short TTL expires valid predictions before their store events arrive. A long TTL keeps failed predictions eligible for routing. Measure routing-to-confirmation latency for successful predictions. Choose the base TTL above the intended percentile plus ingestion and scheduling slack. For example, if p99 confirmation latency is 1.35 seconds and measured collector slack is 0.25 seconds, a 2-second TTL leaves 0.40 seconds of margin. Do not raise TTL from confirmation rate alone. First check whether missing confirmations come from failed requests, pod loss, hash mismatch, or dropped events. Plot confirmation latency by model server, pod, request class, and prefix length. One slow endpoint can make a global TTL look too short. Also measure false-warm cost. Count follower requests routed by a speculative entry that later expires unconfirmed, then record their added prefill tokens and TTFT. The best TTL balances two losses: valid predictions that expire too early and invalid predictions that remain routable too long. ## Make the dashboard decision-ready Start with cohort volume, mature confirmation rate, expiry rate, pending count, and p50, p95, and p99 confirmation latency. Add speculative references, follower routes, server-observed cached tokens, and a no-speculation holdback. This separates event accuracy from workload benefit. Break down the metrics by endpoint and index generation. A pod restart should not let a new cache instance confirm predictions from the previous generation. Alert on sustained routing with no events, growing pending cohorts, a sharp confirmation-rate drop, and false-warm routes that add material prefill work. Keep a minimum volume threshold. A fall from 100 percent to zero after one prediction is not a useful operational signal. Review the numerator and denominator whenever the implementation changes its block annotations, TTL behavior, replay path, or deduplication. Prefix cache speculative index confirmation rate is one layer of the answer. It tells you whether predictions become cache facts. Only downstream cached-token and latency data shows whether those facts helped. --- This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included. Browse the public content map: https://fanout.sh/sitemap.md