--- title: "Prefix cache speculative index TTL" description: "Size a prefix-cache speculative index TTL from route-to-event latency, confirmation failures, and the cost of routing on predicted KV state." canonical_url: "https://fanout.sh/blog/prefix-cache-speculative-index-ttl" md_url: "https://fanout.sh/blog/prefix-cache-speculative-index-ttl.md" last_updated: "2026-10-22" access: "public" --- # Prefix cache speculative index TTL Size a prefix-cache speculative index TTL from route-to-event latency, confirmation failures, and the cost of routing on predicted KV state. - Author: Suraj Gaud - Published: 2026-10-22 - Track: Inference engineering - Access: Public - Tags: prefix caching, llm-d, KV cache, inference routing, TTL, KV events Prefix cache speculative index TTL is the time a router trusts predicted KV-cache state before a model-server event confirms that the blocks exist. Too short, and back-to-back requests miss the prediction before the `BlockStored` event arrives. Too long, and a failed request can leave a false warm-route signal in the index. The useful value comes from the route-to-event latency distribution and the cost of stale predictions, not from request latency as a whole. ## Prefix cache speculative index TTL The[llm-d KV-cache indexer](https://github.com/llm-d/llm-d/blob/main/docs/architecture/kv-management/kv-indexer.md)subscribes to model-server KV events and keeps a near-real-time view of blocks resident on each pod. An event cannot describe a request before that request reaches a model server and stores cache blocks. A second request with the same prefix may reach the router during that gap. With speculative indexing enabled, the router inserts predicted entries for the selected pod immediately after the routing decision. Later requests can match those entries until a confirming `BlockStored` event arrives or the speculative entry expires. The documented default TTL is two seconds. Under prefill/decode disaggregation, llm-d also predicts the selected prefill pod. The TTL therefore covers the event path for the placement that is expected to create the reusable state. ## Measure the interval that TTL covers Start the clock when the router commits the endpoint choice and creates the speculative entry. Stop it when the router's index processes the matching `BlockStored` event. Model execution, event publication, transport, subscriber backlog, and index processing all sit inside this interval. Do not use end-to-end request latency. Decode may continue long after the first reusable prefix blocks are stored, and client network time does not decide when the index becomes truthful. Tag the route decision and confirmation with a request-safe correlation identifier. Record latency without putting prompt text, token IDs, or cache hashes into broad metrics labels. The[precise routing guide](https://github.com/llm-d/llm-d/blob/main/guides/precise-prefix-cache-routing/README.md)uses requests three seconds apart in its verification flow. That interval is intentionally past the router's two-second speculative TTL, so the test routes on confirmed KV events from the prior request rather than on a lingering prediction. Measure this distribution by pod, model, request shape, and router replica. One cluster-wide average can hide a slow subscriber or a model class with later block publication. ## Choose a percentile and add queue slack Let Lq be a high percentile of route-to-confirmation latency. Let J be extra slack for subscriber queue growth, scheduling jitter, and measurement resolution. A starting TTL is: T = Lq + J The percentile is a policy choice. A p95 target accepts more premature expiry than p99, while a maximum can be dominated by a broken or permanently unconfirmed request. Suppose measured confirmation latency is 180 ms at p50, 620 ms at p95, and 1.15 seconds at p99. If subscriber backlog during bursts adds up to 250 ms and the team keeps 100 ms for timing slack, a p99-based TTL is 1.15 plus 0.25 plus 0.10, or 1.50 seconds. Round only after the arithmetic. A two-second setting would leave 500 ms of additional room in this example. Whether that room is worth keeping depends on failed predictions. Recompute Lq after changing event transport, router concurrency, model-server version, block size, or prefill placement. Those changes can move the interval that TTL is supposed to cover. ## Price a failed prediction A route decision does not guarantee that the selected pod will publish the expected blocks. The request can be cancelled, rejected, fail during prefill, restart the pod, or use a cache identity that differs from the router's prediction. Until expiry, a failed speculative entry can make later requests believe a prefix is warm on that pod. The cost is not corrupt model state, because the model server still owns cache truth. The cost is a bad placement decision. The later request may reach a pod without the predicted blocks and pay ordinary prefill after being routed for a hit that did not exist. Let request rate be lambda, failed-prediction probability be p, and TTL be T. A rough steady-state count of live failed predictions is lambda times p times T. At 120 routed requests per second, a 0.4% failed-prediction rate, and a 1.5-second TTL, that is 120 times 0.004 times 1.5, or 0.72 requests' worth of live failed predictions. At five seconds, the same estimate becomes 2.4. This is not a cache-block count, but it makes the stale exposure grow visibly with TTL. Weight that exposure by the extra uncached prefill tokens and the warm-route preference it can distort. A false prediction for a 64-token prefix is not equal to one for 8,192 tokens. ## Separate early expiry from stale expiry Two counters explain opposite TTL failures. Early expiry occurs when a speculative entry expires and a matching confirmation arrives shortly afterward. It indicates that the TTL did not cover the real event path. Stale expiry occurs when the TTL ends without any matching confirmation. It indicates a failed prediction, lost event, identity mismatch, or event pipeline gap. Track confirmation latency for successful predictions, expired-unconfirmed entries, and confirmations that arrive after expiry. Also record later requests that matched a speculative entry and whether the selected pod produced the expected cache hit. That is the outcome the TTL exists to improve. Fanout's[prefix-cache hash guide](/blog/prefix-cache-hash-collision-probability)explains why the index key must represent the same token and parent-hash chain as the model server. A TTL cannot repair key disagreement. It only limits how long a prediction remains eligible while confirmation is absent. ## Account for event gaps and replay The[llm-d precise routing guide](https://github.com/llm-d/llm-d/blob/main/guides/precise-prefix-cache-routing/README.md)describes per-pod event subscriptions. It also provides a replay path for rebuilding index state after a router restart. Speculative TTL and replay solve different gaps. TTL covers the short interval between one routing decision and its event. Replay repairs missed historical state after connection or process loss. If the event subscriber is disconnected, extending TTL does not make the index precise. It keeps predictions alive while confirmations cannot arrive. Pause speculative trust or mark its outcome unknown when event freshness exceeds the operational limit. Do not stretch a two-second mechanism into a substitute for subscriber health. Fanout's[PagedAttention explainer](/blog/pagedattention-vllm-kv-cache)covers the block table beneath prefix reuse. A router index is a placement view of that state, not the allocator itself. Monitor event sequence continuity, replay completion, subscriber queue depth, and time since the last event from each pod beside speculative metrics. ## Validate with paired request gaps Build a test from two requests that share a long exact prefix and differ only in a short suffix. Send pairs at gaps below p50, near p95, near the chosen TTL, and just beyond it. Verify whether each second request routes on a prediction, a confirmed event, or no indexed match. Repeat with successful prefill, client cancellation, forced prefill failure, pod restart, and delayed event delivery. The success case should replace prediction with confirmation before expiry. Failure cases should lose the prediction at TTL and must not look confirmed. Compare TTFT, uncached prefill tokens, endpoint choice, and stale-prediction count. A higher speculative match rate is not useful if false warm routes erase the saved work. Keep block size and cache identity fixed during the TTL sweep. Otherwise, changes in matched prefix length can be mistaken for timing effects. ## Set the control loop Start with the documented two-second default when there is no measured distribution. Treat it as a safe initial hypothesis, not proof for every cluster. An open[llm-d dynamic-TTL proposal](https://github.com/llm-d/llm-d-kv-cache/issues/544)would track last route, last event, event rate, and speculative confirmation rate per endpoint. It proposes longer TTLs when events are delayed and weighted scoring for unconfirmed entries. Treat that behavior as proposed until the deployed router version implements and enables it. After collecting enough confirmations, choose the latency percentile the service wants to cover and add explicit slack. Cap the result with the maximum stale window the routing SLO permits. Alert when late confirmations cross the accepted rate, unconfirmed expiries rise, or subscriber backlog consumes most of the slack. Change TTL slowly. A shorter value reduces stale exposure but can split back-to-back requests across pods. A longer value closes that blind spot while trusting failures for longer. Set the TTL to cover measured route-to-event latency while limiting how long an unconfirmed prediction can affect routing. Handle event-pipeline failure separately. --- This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included. Browse the public content map: https://fanout.sh/sitemap.md