--- title: "Hybrid model cache: state vs KV bytes" description: "Calculate hybrid-model transfer payload from attention KV, recurrent state, padding, tensor parallelism, and missing cache blocks." canonical_url: "https://fanout.sh/blog/hybrid-model-state-vs-kv-transfer-bytes" md_url: "https://fanout.sh/blog/hybrid-model-state-vs-kv-transfer-bytes.md" last_updated: "2026-10-14" access: "public" --- # Hybrid model cache: state vs KV bytes Calculate hybrid-model transfer payload from attention KV, recurrent state, padding, tensor parallelism, and missing cache blocks. - Author: Suraj Gaud - Published: 2026-10-14 - Track: Inference engineering - Access: Public - Tags: hybrid models, KV cache, Mamba, state transfer, disaggregated inference, NIXL, vLLM Hybrid model cache transfer moves two kinds of state: token-growing attention KV and the recurrent data needed to resume one sequence. Attention KV grows with the transferred prefix. A recurrent layer can continue from a fixed-size state snapshot, but the serving system may retain and move several snapshots for cacheable boundaries. Estimate attention bytes per token times missing tokens, then add recurrent-state bytes for the boundaries the decoder must receive. Physical allocator pages are not automatically network payload. ## Hybrid model cache transfer state vs KV bytes The[vLLM hybrid disaggregation design](https://vllm-project.github.io/2026/04/21/hybrid-ssm-disagg.html)starts from two incompatible cache shapes. Full-attention layers store K and V vectors for token blocks. Mamba layers store convolution and state-space tensors that summarize a processed prefix. vLLM exposes two descriptor views over shared cache memory. Attention descriptors point to K and V blocks. Mamba descriptors point to the actual convolution and SSM regions inside the same allocation. A decode worker pulls both through one asynchronous NIXL operation. The transfer is one request from the decoder's perspective, but it can contain many descriptors and two different growth rules. Fanout's[hybrid memory comparison](/blog/hybrid-kv-cache-vs-recurrent-state-memory)covers resident GPU memory. Transfer sizing asks a narrower question: which missing regions cross the link for this handoff? ## Start with logical payload, not page size Let B be attention KV bytes per token after summing the attention layers owned by the transfer. Let T be the number of missing prefix tokens. Let S be the useful convolution and recurrent-state bytes in one resumable snapshot. Let J be the number of state snapshots or logical state blocks that the decoder pulls. The logical payload is B times T plus S times J. T grows with the uncached prefix. J follows the boundaries materialized and requested by the hybrid cache manager. A common page size can hide that difference. The minimal continuation case may need one current state snapshot, so J is one. Prefix caching can retain state at several boundaries, and a connector may transfer more than one logical block. Use the actual descriptor list to determine J. Do not assume that a fixed-size recurrent state means one fixed-size transfer for every runtime and cache policy. ## Work a short and long prefix example The[PyTorch hybrid-model account](https://pytorch.org/blog/hybrid-models-as-first-class-citizens-in-vllm/)reports about 64 KiB for a 16-token attention block and about 2.57 MiB for one Mamba state on NVIDIA Nemotron Nano 12B v2. That attention figure is about 4 KiB per token. Treat it as a model-specific example, not a constant for all hybrid architectures. For a 512-token missing prefix, attention contributes about 2 MiB. One 2.57 MiB recurrent snapshot makes the logical payload about 4.57 MiB. For 4,096 missing tokens, attention contributes about 16 MiB. With the same one snapshot, the payload becomes about 18.57 MiB. At 32,768 missing tokens, attention contributes about 128 MiB. The fixed snapshot is now a small part of the roughly 130.57 MiB total. The crossover occurs at S divided by B. Here, 2.57 MiB divided by 4 KiB per token is about 658 tokens. Below that point, the recurrent snapshot dominates useful bytes. Above it, attention KV dominates. The crossover changes if the connector transfers several state blocks instead of one. ## Convert bytes into a transfer deadline Payload alone does not predict handoff time. A simple link model is fixed posting and completion overhead plus payload divided by effective bandwidth. Suppose effective bandwidth is 20 GB/s and the fixed cost is 80 microseconds. The 4.57 MiB short-prefix payload takes about 0.24 ms on the link, or about 0.32 ms after fixed cost. The 130.57 MiB long-prefix payload takes about 6.85 ms on the link, or about 6.93 ms after fixed cost. These calculations use effective bandwidth, not the interconnect's peak label. Contention, descriptor count, memory layout, and transport protocol lower the delivered rate. The[vLLM NIXL metrics reference](https://github.com/vllm-project/vllm/blob/main/docs/features/nixl_connector_usage.md)exposes transfer time, post time, bytes, and descriptor count separately. That split matters. A small payload with hundreds of fragments can spend more time posting descriptors than moving bytes. A large contiguous payload can show the opposite profile. ## Do not send allocator padding vLLM's[hybrid cache manager](https://docs.vllm.ai/en/stable/design/hybrid_kv_cache_manager/)uses one physical page size across attention and recurrent cache groups. It can increase the attention block size until the attention page covers the Mamba state, then pad the Mamba view to the common page size. That padding simplifies allocation. It does not represent model state and should not be counted as useful network payload. The hybrid NIXL implementation sizes Mamba descriptors to convolution plus SSM bytes rather than the padded HMA page. In the vLLM Nemotron Super 120B BF16 example, skipping padding removed about 50 MB of transfer per request. The FP8 layout in that experiment had little padding because its attention page was smaller. Compare three values during validation: logical state bytes, allocator-reserved bytes, and bytes reported by the transport. A transfer close to reserved bytes may be carrying padding or staging copies. Fanout's[KV-cache memory formula](/blog/kv-cache-formula-llm-inference-memory)supplies the attention term. The connector trace reveals whether the network follows that logical formula. ## Tensor parallelism changes per-rank traffic A tensor-parallel decoder should receive only the shard each rank owns when the layout and connector support that split. The vLLM design uses a dimension-major Mamba convolution layout so x, B, and C subregions are contiguous. A decode rank can read its slice directly instead of pulling the full state and reshuffling it. For decode TP4, each rank can move roughly one quarter of a shardable state rather than the entire state. Aggregate cluster bytes and critical-path bytes are different measurements. If four ranks transfer in parallel, handoff latency follows the slowest rank and shared-link contention. Summing all rank bytes and dividing by one rank's bandwidth overstates one case and understates another. Record bytes, descriptors, and completion time per rank. Then aggregate them only for capacity planning on the shared fabric. Record compatibility in the worksheet too. Cache dtype, layout, tensor-parallel shape, pipeline parallelism, and hybrid allocator support can restrict which direct transfer is valid. Do not infer support from arithmetic. Verify the exact vLLM and connector version before treating a hypothetical shard split as a deployable configuration. ## Missing blocks matter more than prompt length A 32K prompt does not always cause a 32K-token transfer. The decoder may already own part of the prefix, or routing may send the request to a worker with no matching cache. Replace total prompt length with missing tokens T. For each cache group, list the block IDs the decoder requests and the blocks it already has. The hybrid hit is limited by the state needed at one executable boundary. Attention KV beyond the recurrent boundary may not be immediately usable if the recurrent group cannot resume there. Conversely, a recurrent snapshot without the matching attention blocks does not reconstruct full-attention history. Count only the common continuation state when estimating saved prefill. A transport can move many bytes successfully while still delivering a boundary that forces recomputation. Measure transferred tokens, reused tokens, and fresh prefill tokens together. Transfer efficiency is not the same as cache usefulness. ## Compare transfer with recomputation Disaggregation saves work only when moving state and scheduling decode is cheaper than recomputing the missing prefix on the destination. Let X be measured transfer time, including post and completion. Let R be destination prefill time for the same missing prefix and boundary. Transfer has a latency advantage when X is below R. Add queueing and coordination time to both sides before using the comparison for routing. Short prefixes can surprise either way. The recurrent snapshot may dominate bytes, while a fast prefiller could recompute a few hundred tokens before the transfer finishes. Long prefixes usually increase the avoided prefill work, but they also increase attention payload. The crossover depends on prefill throughput and delivered network bandwidth, not context length alone. Fanout's[prefill versus decode guide](/blog/prefill-vs-decode-llm-inference)explains why the saved work is compute-heavy even when decode remains memory-bound. Run the comparison at the p50, p95, and maximum missing-prefix lengths. A policy based on an average can route tail requests into a saturated link. ## Instrument the production handoff Log missing attention tokens, state blocks, logical bytes by cache type, transferred bytes, descriptor count, post time, and end-to-end transfer time. Add effective bandwidth as transferred bytes divided by transfer time. Keep it beside the physical fabric and route so topology changes are visible. Record allocator page bytes and staging-buffer bytes separately. Neither should silently enter the payload numerator. Track expired KV requests. A cheap transfer estimate is irrelevant if the prefiller releases blocks before the decoder finishes reading them. Finally, compare output correctness with a colocated control. State layout or boundary errors can preserve plausible latency while resuming from the wrong data. Size hybrid model cache transfer state vs KV bytes from the descriptors, not the allocator's page labels. Count missing attention tokens, the exact recurrent blocks required to resume, and only the bytes each rank owns. Then test whether moving that state beats recomputing it under the actual link and queue. --- This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included. Browse the public content map: https://fanout.sh/sitemap.md