--- title: "Kimi K3 architecture: KDA, MLA and MoE" description: "A source-led guide to Kimi K3’s hybrid attention, Attention Residuals, Stable LatentMoE, cache design, and deployment tradeoffs." canonical_url: "https://fanout.sh/blog/kimi-k3-architecture-kda-mla-moe" md_url: "https://fanout.sh/blog/kimi-k3-architecture-kda-mla-moe.md" last_updated: "2026-07-29" access: "public" --- # Kimi K3 architecture: KDA, MLA and MoE A source-led guide to Kimi K3’s hybrid attention, Attention Residuals, Stable LatentMoE, cache design, and deployment tradeoffs. - Author: Suraj Gaud - Published: 2026-07-29 - Track: Inference engineering - Tags: Kimi K3, Kimi Delta Attention, mixture of experts, LLM architecture, long context Kimi K3 is easy to summarize badly. Calling it a 2.8-trillion-parameter model is accurate, but it hides the engineering choices that make the number possible. The useful story is about information flow. K3 changes how information moves across tokens, across layers, and across experts. It also forces the serving stack to manage two different kinds of cache. This guide follows the[official technical report](https://github.com/MoonshotAI/Kimi-K3/blob/main/k3_tech_report.pdf), not the launch-day benchmark race. ## The architecture in one pass The[official model summary](https://github.com/MoonshotAI/Kimi-K3)describes a native multimodal mixture-of-experts model with: - 2.8 trillion total parameters - 104 billion activated parameters - 93 layers - 69 Kimi Delta Attention layers - 24 Gated Multi-Head Latent Attention layers - 896 routed experts, with 16 selected per token - Two shared experts - A one-million-token context window - MXFP4 expert weights and MXFP8 activations after quantization-aware post-training Those numbers describe three separate scaling decisions. KDA handles most token mixing with a recurrent state. Gated MLA periodically restores global softmax attention. Attention Residuals select useful representations from earlier depth. Stable LatentMoE adds width without running every expert for every token. MoonViT-V2 supplies the visual pathway. ## Sequence: three KDA layers, then one Gated MLA layer K3 repeats a simple rhythm through most of its backbone: three KDA layers followed by one Gated MLA layer. A final Gated MLA layer ensures the last attention operation is global. That 3:1 ratio matters. Full attention can compare every token with every earlier token, but its work and cache grow with sequence length. Linear attention compresses the past into a fixed-size recurrent state. The state is cheaper to carry through a long sequence, but compression creates a harder question: what should be retained, updated, or forgotten? [Kimi Linear](https://arxiv.org/abs/2510.26692)introduced KDA as a finer-grained extension of Gated DeltaNet. Its experiments reported lower KV-cache use and faster decoding than full MLA at very long context. K3 does not bet the entire model on that compressed state. It interleaves exact global attention so the model still has periodic access to the full token history. This is a practical hybrid, not a claim that linear attention has made ordinary attention obsolete. ## KDA is a writable state, not a smaller KV cache In softmax attention, earlier tokens remain available as keys and values. During decoding, the KV cache keeps those tensors so the model does not recompute the whole prefix at every new token. KDA uses a different object. It updates a recurrent matrix as tokens arrive. A query reads from the current matrix rather than scanning a growing list of token-level keys and values. The delta rule makes the state writable. It first measures what the state already associates with a key, then writes the correction between the old value and the new value. [DeltaNet](https://arxiv.org/abs/2406.06484)showed why this targeted update helps associative recall.[Gated DeltaNet](https://arxiv.org/abs/2412.06464)added a learned decay gate so stale state could be erased more quickly. KDA makes that decay channel-specific. One part of the state can remain stable while another fades quickly enough to accept an update. Fanout’s[How AI remembers Lab](/labs/how-ai-remembers)turns that progression into a small collision experiment. It moves from an exact notebook to an additive summary, a delta update, a forget gate, KDA, and K3’s hybrid. The Lab is deliberately smaller than the real architecture. Its job is to make the update rule visible, not to simulate a 2.8-trillion-parameter model. ## Gated MLA is the exact-attention checkpoint Multi-Head Latent Attention compresses keys and values into a lower-dimensional latent representation. It still performs global softmax attention, but with a more compact cache than conventional multi-head attention. K3 adds a learned gate to the MLA output. The gate controls how much of the global-attention result enters the residual stream at each token. The report frames KDA as efficient long-sequence mixing and Gated MLA as selective high-capacity attention. That division is more useful than saying one mechanism is fast and the other is accurate. KDA carries a compact evolving state. MLA can recover exact token-to-token interactions that a fixed state may blur. ## Depth: Attention Residuals choose what to reuse A conventional residual network passes one accumulated stream from layer to layer. Information from early layers remains present only after being repeatedly transformed and added to. [Attention Residuals](https://arxiv.org/abs/2603.15031)changes that path. Each block can assign data-dependent weights to the embedding and preceding block outputs. The model can retrieve an earlier representation instead of depending only on the latest accumulated residual. K3 uses a block form to control memory cost. It keeps summaries at block boundaries and combines them with the current block’s partial state. This is a second kind of attention, but it operates across network depth rather than across token positions. The architecture therefore has two retrieval axes. KDA and MLA mix information across the sequence. Attention Residuals mix representations across layers. ## Width: Stable LatentMoE routes through 16 of 896 experts K3 expands the expert pool from K2’s 384 routed experts to 896. Each token selects 16 routed experts, alongside two shared experts. The model does not run all 2.8 trillion parameters for every token. The official active-parameter count is 104 billion. That sparsity saves arithmetic, but it creates routing and communication pressure. A badly balanced router can overload a few experts while leaving others underused. Stable LatentMoE addresses this with a latent projection, bounded activations, and Quantile Balancing. Quantile Balancing derives each expert’s routing bias from the score distribution needed to reach a target load. It avoids a fixed bias step that can adapt too slowly or oscillate. The result is not “free capacity.” Expert weights still have to live somewhere, and selected experts have to exchange activations across devices. This is why the[official launch note](https://www.kimi.com/blog/kimi-k3)recommends supernodes with at least 64 accelerators for deployment. ## Serving K3 means managing two cache lifecycles The hybrid attention pattern produces two different cache shapes. The MLA KV cache grows with the number of cached tokens. The KDA recurrent state stays fixed in size for a request, but it must represent the exact prefix boundary being reused. The report describes a unified paged pool for both. MLA entries use token-level pages, while KDA checkpoints are stored at selected prefix boundaries. This creates a subtle prefix-caching problem. Reusing an MLA prefix without the matching KDA state would restore two mechanisms to different points in the conversation. K3 separates physical allocation from hash granularity. A large physical page can contain smaller hash blocks, while KDA checkpoints are saved only at reusable endpoints. The serving lesson is broader than K3. A cache is valid only when every piece of model state agrees about which prefix it represents. The[KV-cache Lab](/labs/kv-cache)shows how ordinary token-level KV memory grows with context and concurrency. K3 adds a fixed recurrent state beside that growing cache. ## One million tokens is a systems claim too A large context limit does not guarantee that every token is equally useful, affordable, or fast. The report describes cache-aware routing, budget-based admission control, and separate request classes for short and ultra-long traffic. That is the operational side of long context. A burst of million-token requests can consume enough prefill work and cache capacity to damage time to first token for small requests. K3’s fleet scheduler routes a session toward the cluster holding its prefix cache. It also assigns long-context traffic its own resource budget so it cannot consume the entire serving pool. These are not optional details around the model. They are part of making the advertised context length usable in a shared service. Fanout’s upcoming[inference engineering track](/inference-eng)treats this boundary as the main subject: model architecture, cache layout, admission control, latency, and cost belong in one design. ## Read the benchmarks with their harnesses attached Moonshot reports strong coding, agentic, reasoning, and multimodal results. The same report also says K3 trails the strongest proprietary systems overall. The useful caution is in the footnotes. Different models sometimes use different agent harnesses, reasoning settings, fallback behavior, or hardware. A benchmark row measures the model and that evaluation setup together. It does not isolate architecture from post-training, tools, prompts, or inference policy. K3 is worth studying without turning every table entry into an independent fact. The open weights and technical report make mechanism-level inspection possible. ## What to carry forward K3’s architecture can be read as three retrieval decisions: - Across tokens, use KDA most of the time and Gated MLA periodically - Across layers, use Attention Residuals to select earlier representations - Across width, route each token through a sparse subset of experts The serving system then has to preserve those decisions across cache reuse, devices, and requests. Start with the[official repository](https://github.com/MoonshotAI/Kimi-K3), then use the[interactive memory Lab](/labs/how-ai-remembers)to make the KDA progression concrete. For the cache side, continue with Fanout’s[PagedAttention Daily note](/daily/2026-07-21-pagedattention)and the[KV-cache calculator](/labs/kv-cache). --- This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included. Browse the public content map: https://fanout.sh/sitemap.md