Prefix cache salt granularity explained

Prefix cache salt granularity sounds like one setting, but it combines two separate decisions. Salt scope chooses who may share cached prefixes. Match granularity chooses where a hit may end.

A per-tenant salt can preserve reuse inside each tenant while blocking reuse across tenants. A 32-token prefix match unit can still expose hit boundaries every 32 tokens inside that salt namespace.

Changing one does not tune the other. Treat security scope and token boundary as separate rows in the serving design.

Prefix cache salt granularity has two axes

vLLM hashes prefix blocks so a later request with the same token prefix can reuse stored KV state.

Its automatic prefix caching design adds an optional cache_salt to the first block's hash. Only requests with the same salt can match that chain.

The salt is a namespace barrier. It decides which requests are eligible to see the same cache keys.

The prefix match unit is a token interval. Current vLLM cache configuration defines it as the finest token boundary where a prefix hit may land.

One tenant can use a 32-token match unit, and another tenant can use the same unit, without their entries matching when their salts differ.

Fanout's prefix match unit guide covers the storage and scheduler boundaries behind that token interval.

One first-block salt protects the chain

The cache key for a later block includes the hash of the previous block. That parent relationship turns the prefix into a chain rather than a set of independent chunks.

vLLM injects cache_salt into the first block. Every later hash depends on that salted parent, so the namespace difference propagates through the full reusable prefix.

Two requests with identical tokens and different salts therefore produce different first-block hashes and different descendants.

The original vLLM cache-salting RFC describes the goal as preserving reuse among requests that share a trust group while separating other callers.

It discusses organization-level and user-level scope. The current single-salt request field creates one barrier at the root rather than several barriers inside one prompt.

That distinction matters for mixed prompts. A globally shared system message and a tenant-private document cannot use different salt scopes inside the same root-salted chain.

Do not infer that a smaller prefix match unit creates another security boundary. It creates more candidate hash positions under the same salt.

Work through the reuse arithmetic

Assume 100 tenants each send 20 requests with the same 2,000-token prefix. The cache retains every entry, all requests reach the same cache, and nothing is evicted.

With one global namespace, the first request fills the prefix and the remaining 1,999 requests can reuse it. This maximizes sharing but crosses tenant boundaries.

With one stable salt per tenant, each tenant pays one cold fill. That is 100 fills and 1,900 possible reuses.

With a fresh random salt per request, all 2,000 requests occupy separate namespaces. Cross-request reuse falls to zero even though every token is identical.

These are idealized opportunities, not expected hit rates. Routing, concurrent cold starts, capacity, eviction, and serialization differences can turn an eligible reuse into a miss.

The arithmetic shows what salt scope removes before those operational losses begin. Per-tenant scope costs 99 extra cold fills in exchange for isolation between the 100 tenants.

Per-request scope costs 1,999 extra fills. Use it only when the threat boundary requires every request to be isolated from every other request.

Match units change the reusable tail

Now hold the tenant salt fixed and compare token boundaries.

Suppose two requests share 1,500 tokens. A 1,024-token match boundary recognizes only the first 1,024, so the second request recomputes 476 shared tokens.

A compatible 32-token match unit recognizes floor of 1,500 divided by 32, times 32. That is 1,472 reusable tokens and 28 recomputed tokens.

Both requests still need the same salt. Finer matching does not let one tenant reach another tenant's cache entries.

Likewise, changing the salt does not change 1,472 into another token boundary. It only decides whether that existing boundary is eligible to match.

Fanout's PagedAttention explainer covers the physical block table beneath these content hashes.

On hybrid models, the match unit may be smaller than a large recurrent-state block. The engine still needs compatible cache managers and safe state at the finer boundary.

Measure recovered prompt tokens alongside hit count. Two requests can both report a hit while one reuses 1,024 tokens and the other reuses 1,472.

Choose salt scope from the trust boundary

The current vLLM security guidance recommends a per-tenant secret for multi-tenant deployments and treats cache_salt as defense in depth.

That makes tenant scope the starting point when one organization must not learn whether another organization's prompt was cached.

User scope is stricter. It prevents reuse across users in the same tenant, which may be necessary when those users are separate principals.

Session scope is stricter again. It keeps a growing conversation reusable within one session while preventing a second session from sharing its cache.

Request scope removes almost all repeated-prefix reuse unless retries deliberately reuse the same value. It can be appropriate for highly isolated work, but it should be a conscious capacity choice.

A service-wide salt is close to no isolation between callers who share that value. It protects only against parties that cannot obtain the salt and cannot send through a path that supplies it.

Keep the salt out of logs, traces, client-visible errors, and metrics labels. Anyone who can submit the same salt joins that cache namespace.

The public GitHub advisory for CVE-2025-46570 explains the timing risk that motivated isolating prefix-cache reuse.

Rotation creates a planned cold start

Rotating a salt changes the first hash, so the old and new cache chains no longer match even when the prompt tokens are identical.

For the 100-tenant example, rotating every tenant at once creates at least 100 new cold fills for the shared prefix under the idealized assumptions.

Rotating one tenant at a time spreads those fills across the rollout. It also leaves two active salt generations during the migration unless requests switch atomically.

Do not send both old and new salts with one request unless the serving API and security design explicitly support multiple namespaces. Current root salting uses one cache_salt value per request.

Track the salt generation as internal control-plane state, not as a prompt field. The model does not need the value, and the prompt body should not contain it.

Measure cold TTFT and prompt-token recomputation during rotation. A hit-rate chart alone may hide that the missed prefixes are unusually long.

Set a rotation policy from the secret's exposure risk and operational recovery plan. A faster schedule is not automatically safer if it causes teams to reuse predictable values or disable salting.

Capacity splits with every namespace

Salted copies of the same token prefix are separate cache entries. If 100 tenants all use the same 2,000-token system prompt, the cache may retain 100 isolated copies.

That duplication is the memory price of the chosen boundary. It can evict unrelated prefixes sooner even when the per-tenant hit rate is high.

Estimate the footprint by multiplying cached tokens per repeated prefix by the number of active salt namespaces, then by KV bytes per token.

Fanout's KV-cache memory formula supplies the bytes-per-token term for a model, precision, and parallel layout.

The estimate needs a retention model. If only ten tenants are active during the cache lifetime, pricing all 100 overstates the resident copies.

Record active namespaces, resident salted prefixes, evictions, reusable tokens, and KV-cache occupancy. Those measurements reveal whether isolation is reducing useful cache density.

Do not merge tenants merely to improve hit rate. A lower memory bill is not evidence that the wider sharing boundary satisfies the security model.

Test the two knobs independently

First fix the prefix match unit and replay the same trace under tenant, user, session, and request salt scopes that the threat model permits.

Compare eligible reuse, matched tokens, cold TTFT, evictions, and cache occupancy. This isolates the namespace decision.

Then fix the salt policy and sweep valid prefix match units. Compare extra recovered tokens with hash lookup overhead and scheduler constraints.

Use byte-identical serialized prompts. Prefix caching follows token IDs and hash chains, so semantically equivalent JSON can miss if field order or whitespace changes tokenization.

Keep request routing stable. A request sent to a worker without the relevant local or shared cache can miss even when its salt and tokens match.

Document both settings in the deployment review: who shares a salt, how it rotates, and the smallest token boundary the engine enabled in practice.

Prefix cache salt granularity becomes clear once the axes stay separate. Salt scope defines the trust group. Match granularity defines how much of an eligible prefix the engine can reuse.