Multi-barrier prefix cache salting

Multi-barrier prefix cache salting tries to preserve shared public prefixes while splitting the cache at later trust boundaries. One request can cross a global barrier, an organization barrier, and a user barrier.

The idea fixes a real weakness in flat salting. A per-user root salt protects the whole request but destroys safe reuse of a common system prompt.

A shared organization salt keeps reuse but may expose user-private suffixes inside that organization.

Today, multi-barrier salting is not shipped in vLLM. The current API has one request-level cache_salt. Token-range barriers remain an open proposal and pull request as of October 10, 2026.

Multi-barrier prefix cache salting changes the hash chain

Automatic prefix caching identifies a block from its tokens and the hash of its parent block. A salt inserted into one block changes that block's hash and every descendant hash.

The vLLM prefix-caching design describes the shipped behavior. A request salt is included in the first block, so only requests with the same salt can reuse the resulting chain.

A multi-barrier design inserts another salt at a later token boundary. Blocks before that boundary keep their earlier identity. Blocks after it inherit the new salt through the parent hash.

The result is a tree of reuse scopes. A global prefix can be shared by everyone. An organization document can branch under an organization salt. A private conversation can branch again under a user salt.

The barriers are not independent caches. They are points where the hash lineage changes. A mismatch at an earlier barrier prevents reuse of every later block, even if a later salt happens to match.

Fanout's prefix-cache hash collision calculation covers the digest width and adversarial cases behind that content identity.

A single root barrier forces one scope

Suppose every request contains three stable segments: a public system and tool prefix, an organization document, and a user-private history.

With one per-user root salt, all three segments are isolated by user. The public and organization segments are recomputed for the first request from every user.

With one per-organization root salt, the public prefix and organization document are reused inside the organization. The private history is also in that same reuse scope, which is too broad if users must be isolated from one another.

With no salt, the cache can reuse matching prefixes across callers. That may be acceptable for a genuinely single-tenant deployment. It is a poor default for an authenticated multi-tenant endpoint.

The original vLLM multi-salt RFC proposes placing salts at message indices so each stable region can inherit the narrowest scope it needs.

Work the reuse arithmetic

Consider 10 organizations with 10 active users each. Every user repeats a request with a 640-token global prefix, a 1,920-token organization document, and 384 tokens of private history.

There are 100 users. With a per-user root salt, the first request from each user fills all 2,944 tokens in that user's cache lineage.

100 × 2,944 = 294,400 cold token computations.

A multi-barrier layout can fill the 640-token global prefix once, the 1,920-token document once per organization, and the 384-token private suffix once per user.

640 + (10 × 1,920) + (100 × 384) = 58,240 cold token computations.

That is 236,160 fewer cold token computations, an 80.2% reduction in this idealized workload.

Treat the 80.2% as accounting, not a throughput promise. It assumes byte-identical prefixes, stable tokenization, cache residency, compatible routing, and repeated requests.

Eviction or a changed document reduces the realized hit rate.

The arithmetic does show why one salt is structurally limiting. Safe reuse exists at more than one trust scope, but a root salt can name only one scope for the entire descendant chain.

Block boundaries decide where a barrier lands

Prefix caches usually hash complete token blocks. A policy may describe a barrier at a message boundary, but the implementation still has to map that boundary onto blocks.

If a 16-token block contains the last 10 tokens of a public message and the first 6 tokens of a private message, sharing that block would cross the trust boundary.

A safe implementation must split, pad, or treat the mixed block as belonging to the narrower scope. Each choice changes reuse or storage.

The worked example uses segment lengths divisible by 16, so it avoids that rounding. Real chat templates rarely cooperate so neatly.

Token-range configuration must also survive template changes. A barrier expressed as a stale token offset can move after a system prompt edit, tokenizer change, or tool-schema update.

Message-aware configuration is easier to reason about, but the engine must resolve it to exact token ranges after rendering and tokenization.

Current vLLM has one shipped barrier

The current request field is cache_salt. It salts the first block, and the altered parent hash separates the rest of the request.

The multi-barrier RFC calls the proposed field cache_salt_map. That name and shape are not a stable public contract.

The associated token-range cache salt pull request is still open as of October 10, 2026. It should be evaluated as work in progress, not configured as an available release feature.

Architecture documents need this distinction. “vLLM supports cache salting” is true for one request-level barrier. Several salt regions in one request are not yet a production feature.

Until the feature lands in a release you have tested, the deployable choices are flat salting, separate engine instances, or accepting a broader reuse scope with explicit risk review.

The gateway owns salt identity

The vLLM security guidance says the gateway should authenticate callers and choose the salt scope.

The engine should not trust an arbitrary caller-supplied tenant label.

Predictable account IDs are also poor salts. vLLM recommends random 256-bit values encoded as 43-character base64 strings. A gateway can map authenticated principals to opaque salts without exposing their construction.

For a future multi-barrier request, the gateway would need to authorize every scope in the chain. A user who belongs to organization A must not be able to request organization B's salt for the document region.

Salt rotation is another policy choice. Rotating a user salt invalidates reuse beneath that barrier without forcing a new global or organization lineage. The old entries remain until eviction unless the cache supports targeted removal.

Salting reduces timing leakage from prefix-cache hits, but vLLM explicitly does not present it as a complete tenant-isolation boundary. Strict isolation still calls for dedicated instances and independent resource limits.

Test security and reuse together

A cache test should send identical prefixes under equal and unequal scope chains. Equal global salts should hit the public blocks. Unequal organization salts should miss the document and every descendant block.

Equal organization salts with unequal user salts should hit through the document, then miss at the private barrier. That transition is the defining behavior of the design.

Measure computed tokens or cache-hit blocks alongside latency. Queueing, batching, and kernel warmup can hide whether the prefix was actually reused.

Then test adversarial identities. Reject unknown organizations, mismatched user memberships, missing barriers, duplicate ranges, and ranges that do not align with the rendered request.

The timing-attack work in Prompt Cache Leakage in LLM APIs is a reminder that cache hits can reveal shared-prefix information. Salt tests need negative cases, not just throughput wins.

Fanout's guide to prefix-cache salt granularity covers the flat choice between global, organization, user, and request scopes. Multi-barrier salting extends that model into one ordered request.

Treat barriers as trust-boundary transitions

Think of the request as a hash chain that narrows as the prompt becomes more private.

Start with content safe for global reuse. Insert an organization barrier before tenant documents. Insert a user barrier before private history. Never widen the scope again later in the chain.

That model can recover substantial reuse compared with a per-user root salt. In the worked example, ideal cold computation falls from 294,400 to 58,240 tokens.

It also adds policy and alignment complexity. The gateway must own opaque identities, the tokenizer must place barriers correctly, and observability must prove both expected hits and required misses.

For current vLLM deployments, keep using the single cache_salt field or stronger process isolation. Track the multi-barrier RFC and pull request, but do not write future API behavior into production configuration yet.