Compare llm-d prefix cache and server hits

The useful llm-d prefix cache index hits vs server hits comparison uses two token shares. One is what the router predicted on the chosen endpoint. The other is what the model server says it actually reused.

Do not divide a router numerator by a server denominator. The built-in pairs can cover different requests, and some tokenizer or disaggregated-serving modes make their token counts describe different work.

Start with each ratio on its own. Compare them only after checking the cohort, token unit, endpoint role, and response path.

Define llm-d prefix cache index hits vs server hits

The llm-d router metrics reference exposes a predicted pair for the chosen endpoint.

Divide `llm_d_epp_prefix_predicted_cached_tokens_sum` by `llm_d_epp_prefix_prompt_tokens_sum`. The result is the share of prompt tokens the router expected the selected endpoint to hold.

The delivered pair is separate. Divide `llm_d_epp_request_cached_tokens_sum` by `llm_d_epp_request_input_tokens_sum` to measure the share the model-server response reported as cached.

For vLLM's own Prometheus counters, divide `vllm:prefix_cache_hits_total` by `vllm:prefix_cache_queries_total` over the same time window and pod set.

Fanout's prefix match unit guide explains why both sides should count only the reusable prefix before the first missing block.

The comparison asks whether the router's view tracks delivered reuse. It does not, by itself, prove that routing caused the hit or improved latency.

Keep each metric pair intact

Prometheus histogram sums make the router ratio straightforward. Sum the predicted cached tokens across the chosen scope, then divide by the matching prompt-token sum from the same observations.

Apply the same rule to the delivered pair. Its cached-token sum and input-token sum come from model-server responses.

Never calculate predicted cached tokens divided by delivered input tokens. A request enters the prediction metrics before forwarding, but enters the delivered metrics only after a response with usage data.

A failed request can appear in the predicted pair and disappear from the delivered pair. So can a response path that omits usage.

The official reference says not to divide across the two pairs for this reason. Compare the two finished percentages, then inspect volume and cohort differences before naming the gap.

Use rate or increase functions over counters where appropriate. Do not compare lifetime totals from a restarted router with lifetime totals from a server that has been running for days.

Work through the aggregate ratios

Assume a ten-minute router window records 480,000 predicted cached tokens over 600,000 prompt tokens.

The router-predicted share is 480,000 divided by 600,000, or 80 percent.

In the same wall-clock window, model-server responses report 348,000 cached tokens over 480,000 input tokens.

The delivered share is 348,000 divided by 480,000, or 72.5 percent. The chart shows a 7.5 percentage-point difference.

Do not label all 7.5 points as stale index error. The server pair covers 120,000 fewer input tokens, so the two aggregates do not describe an identical request cohort.

First graph all four sums. A percentage gap accompanied by a denominator gap needs request-path investigation before cache diagnosis.

Also graph successful responses and missing-usage responses. Those counts explain whether the delivered denominator shrank because the server reused less or because fewer requests produced comparable data.

Build a matched cohort when the gap matters

For a controlled test, keep only successful requests that carry both the router prediction and the server's cached-token result.

Suppose that matched cohort contains 480,000 input tokens. The router predicted 384,000 cached tokens, while the servers delivered 348,000.

Now both shares use the same denominator. Predicted reuse is 80 percent, delivered reuse is 72.5 percent, and the shortfall is 36,000 tokens.

That result still needs a time boundary. A block can exist when the router scores the endpoint and be evicted before the server schedules the request.

Built-in aggregate metrics do not provide a request identifier label, which is a good cardinality choice. Use a bounded load test, sampled traces, or structured logs for the matched analysis.

Do not add raw prompts to the correlation data. Request identifiers, endpoint identity, token counts, and timestamps are enough for this measurement.

The precise routing guide uses repeated shared-prefix requests and per-pod counters as a smoke test.

Check that token units are comparable

The vLLM render backend gives the router the same token IDs used by the model server. Under that path, the predicted and delivered token shares are directly comparable.

The router's zero-configuration estimate backend uses four-byte pseudo-tokens. The metric reference warns that code, CJK text, and chat templates can shift this estimate against server tokens.

A stable ratio gap under the estimate backend may reflect token units rather than missing cache state.

Run a representative prompt sample through the exact render path before setting a tight alert threshold. Record which backend produced the series in the dashboard label or panel description.

Block geometry must agree too. The precise guide tells operators to check the router's `blockSizeTokens` against vLLM `--block-size` or SGLang `--page-size` when requests spread and server hits remain low.

Fanout's prefix-cache hash guide covers the identity chain. A correct token count still misses when model, adapter, salt, or multimodal identity differs.

Respect endpoint roles in disaggregated serving

Prefill and decode can run on different endpoints. The router prediction and the response usage must refer to the same cache owner before their gap means index accuracy.

The llm-d metrics reference says the prediction describes the prefill endpoint for a disaggregated request.

With the default `nixlv2` connector, the returned cached-token count also represents the prefiller, so the two views can be compared.

Other connectors can return the decoder's usage unchanged. In that case, a prefill prediction compared with a decode cached-token result describes different endpoints.

Split dashboards by endpoint role and connector. If the response cannot attribute cached tokens to the prefill side, label the comparison unavailable instead of plotting a misleading gap.

The same caution applies to candidate pools. A prefix held on a decode-only pod can raise an available-cache maximum even though the prefill decision could never choose that pod.

Read the direction of a credible gap

The current llm-d precise prefix-routing guide recommends comparing the router's predicted token share with the servers' hit rate.

If both shares are low, the workload may simply contain little repeated prefix. Check repeated-prefix volume before debugging the event pipeline.

If server reuse is materially higher than the router prediction, the index may be missing blocks. Inspect active subscribers, replay recovery, skipped store events, queue depth, and render failures.

If the router predicts more reuse than the server delivers, check evictions after routing, stale pod generations, dropped removal events, salt changes, and block-size mismatch.

Also check the denominator gap, tokenizer backend, connector, and time alignment. Those measurement errors can produce the same chart shape.

The precise guide notes that `active_subscribers` equals matched pods by construction. A wrong pod selector can still make that number look healthy.

Compare it with the count of model servers actually publishing KV events. Then inspect event rates while routing stays active.

Separate index quality from routing quality

A correct index can still lead to weak delivered reuse if filters remove the warm endpoint or the picker favors load over cache affinity.

llm-d exposes the chosen endpoint's predicted tokens, the best prediction among scored endpoints, and the best available prediction before filtering.

Chosen divided by best predicted measures how much reachable reuse the picker captured. Best predicted divided by best available measures how much survived filtering.

Interpret the second ratio only when the candidate roles are comparable. In disaggregated serving, the pre-filter set can include decode pods that a prefill decision cannot reach.

Keep these routing ratios separate from index-versus-server agreement. One diagnoses selection policy. The other diagnoses whether the router's cache view matches execution.

The broad llm-d architecture pages explain cache-aware routing, but the operator needs this split to decide which component to change.

Connect cache agreement to serving outcomes

A high agreement rate can coexist with poor latency if the shared prefixes are short or the warm endpoint is saturated.

Add matched cached tokens, uncached prompt tokens, TTFT, prefill accelerator time, queue delay, and request success to the same experiment.

Compare against a load-aware baseline without prefix affinity. That separates useful routing from reuse the workload would have received on the chosen pod anyway.

Fanout's goodput and SLO guide shows why a higher throughput count can still miss the latency target that matters.

The recent llm-d 1.0 account reports production cache-hit and TTFT gains.

Those fleet results do not set a threshold for another workload.

Set an alert from your matched baseline. Require enough prompt-token volume that one long request cannot swing the percentage.

Start with the two valid ratios. Verify comparable cohorts and token units, then narrow a real gap through event health, endpoint roles, filter loss, and picker loss.