--- title: "Hybrid KV cache vs recurrent state memory" description: "Calculate when token-growing KV cache overtakes fixed recurrent state in a hybrid LLM, then include concurrency, precision, and allocator padding." canonical_url: "https://fanout.sh/blog/hybrid-kv-cache-vs-recurrent-state-memory" md_url: "https://fanout.sh/blog/hybrid-kv-cache-vs-recurrent-state-memory.md" last_updated: "2026-09-30" access: "public" --- # Hybrid KV cache vs recurrent state memory Calculate when token-growing KV cache overtakes fixed recurrent state in a hybrid LLM, then include concurrency, precision, and allocator padding. - Author: Suraj Gaud - Published: 2026-09-30 - Track: Inference engineering - Access: Public - Tags: hybrid models, KV cache, recurrent state, Mamba, vLLM, GPU memory, LLM inference Hybrid KV cache vs recurrent state memory is a comparison between one cost that grows with context and another that is fixed per live sequence. Attention layers append keys and values for every cached token. Mamba and linear-attention layers keep a recurrent state whose shape does not grow with the prefix, but every concurrent request needs its own copy. The crossover is the recurrent-state bytes per request divided by the attention KV bytes per token. Context decides which pool is larger. Concurrency multiplies both. ## Hybrid KV cache vs recurrent state memory A hybrid model interleaves ordinary attention with recurrent layers. The attention layers preserve token-level history, while the recurrent layers update a compact summary of the prefix. The[PyTorch account of hybrid support in vLLM](https://pytorch.org/blog/hybrid-models-as-first-class-citizens-in-vllm/)gives a concrete serving example. For NVIDIA Nemotron Nano 12B v2, it reports attention KV blocks of about 64 KiB for 16 tokens and a Mamba state of about 2.57 MiB per sequence. One KV block is smaller than one recurrent state. That local comparison is misleading because a long sequence owns many KV blocks but still owns one current recurrent state. The same report says the KV cache at 128K tokens can be nearly 200 times larger than the Mamba state. The fixed cost wins early; the linear cost wins later. Fanout's[Kimi K3 architecture guide](/blog/kimi-k3-architecture-kda-mla-moe)shows why a production model can need both forms of state instead of choosing one globally. ## Derive the crossover context Let S be recurrent-state bytes per live request after summing every recurrent layer. Let B be attention KV bytes per cached token after summing every attention layer. At context length T, the logical request-state budget is S plus B times T. KV and recurrent state are equal when B times T equals S. The crossover length T star is therefore S divided by B. Below T star, recurrent state consumes more memory. Above T star, attention KV does. The model remains hybrid on both sides; only the larger line item changes. For attention, B starts with 2 times L times H times D times Q. L is the number of attention layers, H the KV heads, D the head dimension, and Q the bytes per stored element. The factor 2 accounts for keys and values. Divide by tensor-parallel size only when the deployed backend actually shards those KV heads across the ranks in question. Fanout's[KV-cache memory formula](/blog/kv-cache-formula-llm-inference-memory)develops that calculation for full attention, grouped-query attention, and tensor parallelism. ## Work the reported Nemotron example backward The PyTorch article gives enough information for an approximate crossover without copying private model code. If a 128K-token KV cache is nearly 200 times a 2.57 MiB Mamba state, the KV cache is roughly 514 MiB. Dividing by 131,072 tokens gives about 4.0 KiB per token. Dividing 2.57 MiB by about 4.0 KiB per token puts the equal-memory point near 650 tokens. At 256 tokens, the KV term is roughly 1 MiB while recurrent state remains about 2.57 MiB. At 8,192 tokens, KV is roughly 32 MiB and has become the larger pool. At 128K, the estimate returns about 512 MiB of KV beside the fixed 2.57 MiB state. Small rounding differences explain the article's "nearly 200 times" wording. This is an inference from the published ratio, not an allocation promise. The actual figure depends on cache dtype, tensor parallelism, alignment, padding, and which layers the implementation groups together. Use the same arithmetic on the deployed model's resolved shapes. Architecture names alone do not supply a reliable B or S. ## Concurrency multiplies the fixed state too For Q equal-length live requests, logical state is Q times S plus Q times B times T. The crossover T star does not change when every request has the same length, because Q multiplies both terms. The total bytes still grow directly with Q. Suppose S is 60 MiB per request and B is 20 KiB per token. T star is about 3,072 tokens. At 256 requests, recurrent state alone is 15 GiB. That matches the scale reported for Qwen3.6-35B in the DAMP measurements cited by the open Mamba-state work. At an average context of 2,048 tokens, attention KV adds 10 GiB across those requests. At 8,192 tokens, it adds 40 GiB and overtakes the recurrent pool. Real batches contain unequal contexts. Replace Q times T with the sum of cached tokens across requests, while recurrent state remains Q times S. This is why an average context length can hide an admission failure. A few long sessions raise the KV sum, while every short session still reserves one recurrent state. ## Physical allocation can move the apparent crossover The logical formula assumes each state uses exactly its tensor bytes. Current allocators may need equal-size pages across different layer groups. The[vLLM hybrid KV cache manager design](https://docs.vllm.ai/en/stable/design/hybrid_kv_cache_manager/)describes this constraint for Mamba hybrids. It increases the attention block size until attention bytes can cover a recurrent-state page, then pads the recurrent page so both groups share one physical page size. The design notes that attention block sizes can exceed 400 tokens in some cases. That changes tail waste, prefix-cache granularity, and how much useful KV fits in a nominal memory pool. Logical B times T plus S is still the right model-level budget. Physical bytes can be higher because the allocator rounds both sides to page boundaries. Measure the resolved attention block size, recurrent page bytes, number of groups, and allocated pages. Do not infer them from the default block-size flag. Fanout's[prefix match unit comparison](/blog/prefix-match-unit-vs-kv-cache-block-size)explains why a large physical page does not have to become the prefix lookup unit. ## Precision changes the two terms independently Moving attention KV from BF16 to FP8 roughly halves the raw attention term B when the backend supports that dtype. It does not automatically halve S. Recurrent state may use FP32, FP16, BF16, or a separate quantized representation with different numerical risks. If only B halves, T star doubles. The fixed recurrent state remains dominant for a longer context range even though the model admits more KV tokens. One current[vLLM hybrid-allocation RFC](https://github.com/vllm-project/vllm/issues/55196)reports another complication: a larger non-quantized recurrent page can pin the shared physical page size. In that layout, an FP8 attention tensor can use half the logical bytes without delivering a full two-times increase in pool capacity. Treat that issue as implementation evidence, not a universal result. It covers named models, revisions, and allocator behavior that can change. Recurrent-state quantization needs its own quality test because each generated token reads and rewrites the state. KV entries are normally written once and read many times. The two precisions belong on separate rows in a memory and accuracy worksheet. ## Transfers pay for both state shapes Prefill-decode disaggregation must move the state needed to continue generation on another worker. The[vLLM hybrid SSM transfer design](https://vllm-project.github.io/2026/04/21/hybrid-ssm-disagg.html)stores full-attention pages and complete recurrent snapshots with different descriptor views. Attention transfer bytes grow with the reusable prefix. Recurrent transfer bytes stay near S for one boundary, but another checkpoint or resumed branch needs another exact snapshot. For a short prompt below T star, the recurrent snapshot can dominate transfer time even though the architecture is marketed for cheap long contexts. For a long prompt, KV dominates again. The crossover for transfer may differ from the resident-memory crossover because the transport can skip, compress, or reuse some pages. Record bytes actually sent by state type, not only the common allocator page size. Padding useful for GPU allocation does not become useful network payload. Also record the prefix boundary. A recurrent state represents one exact processed prefix, while token-level KV pages may support more granular reuse. ## Build an admission worksheet from resolved shapes Start with model weights and runtime workspace. The memory left after those fixed server costs is the state pool for live requests. For attention, calculate B from the deployed layer count, KV heads, head dimension, storage dtype, and sharding. Confirm it by measuring cache bytes added per long request. For recurrent layers, sum convolution and SSM state bytes across layers. Confirm S by comparing idle allocation before and after adding one short live sequence. Calculate T star = S/B. Then test contexts below, near, and above that point with several concurrency levels. Compare logical bytes with allocator-reserved bytes. Large gaps point toward page padding, tail waste, duplicated groups, graph capture, or framework reserves rather than the model-state formula. Repeat after changing KV precision, recurrent-state precision, tensor parallelism, or the hybrid allocator. Each change can move one term without moving the other. Finally, profile latency. A smaller state is useful only if its update, cache-read, or transfer kernels keep the service inside its latency target. Use S plus B times T as the first budget for each live sequence. Then compare it with allocator-reserved bytes at the contexts and concurrency the server actually carries. --- This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included. Browse the public content map: https://fanout.sh/sitemap.md