How Kimi K3 Scales: MoE, KDA, and Attention Residuals

Moonshot AI introduced Kimi K3 as a 2.8-trillion-parameter, natively multimodal model with a one-million-token context window. Those numbers make the headline, but they do not explain why the model is technically interesting.

A recent Kimi K3 explainer video organizes the architecture around four ideas: Mixture of Experts, Stable LatentMoE, Kimi Delta Attention, and Attention Residuals. That is a more useful way to examine K3 than treating total parameter count as a direct measure of quality.

The four mechanisms address different scaling problems:

• Mixture of Experts expands model capacity without using every parameter for every token.
• Stable LatentMoE tries to keep extremely sparse expert routing trainable and balanced.
• Kimi Delta Attention reduces the cost of carrying information across long sequences.
• Attention Residuals gives deep layers more control over which earlier representations they reuse.

Together, they show where large-model engineering is moving: not simply toward more parameters, but toward more selective use of parameters, context, and depth.

What 2.8 trillion parameters actually means

In a dense model, every token passes through the same feed-forward parameters. Increasing model size therefore increases the computation required for nearly every token.

A Mixture-of-Experts model changes that relationship. A router sends each token to a small subset of specialized feed-forward networks, or experts. The model can have a very large total capacity while activating only part of it for each token.

Moonshot says K3 routes a token through 16 of 896 experts. That is extreme sparsity: fewer than two percent of the experts participate in a particular routing decision. The 2.8-trillion figure describes the full model, not the amount of computation applied to every token.

This distinction is important, but it does not make the model small. All those weights still need to be stored, distributed, loaded, and coordinated. Expert routing also creates communication and load-balancing costs. Moonshot recommends deployment on supernodes with at least 64 accelerators, which makes the practical point clearly: open weights do not automatically mean local or inexpensive inference.

For an engineering team, total parameters, active parameters, memory footprint, interconnect requirements, and tokens per second are separate measurements. Collapsing them into one model-size number hides the system that must actually serve the model.

Why Stable LatentMoE matters

Sparse routing becomes harder as the number of experts grows. If the router repeatedly favors a small group of experts, those experts become overloaded while much of the model remains undertrained. If routing is forced to be too uniform, the model may lose the specialization that makes MoE useful.

Moonshot describes K3's solution as Stable LatentMoE. Its launch material also identifies two supporting techniques:

• Quantile Balancing derives expert allocation from router-score quantiles instead of relying on heuristic balancing updates and a sensitive tuning parameter.
• Balanced expert-parallel training uses static shapes and removes host synchronization from the critical path, reducing the throughput penalty created by uneven expert traffic.

The important systems lesson is that routing quality and hardware utilization are coupled. An expert-selection mechanism can look mathematically sparse and still run inefficiently if traffic concentrates on the wrong devices.

Moonshot had not yet published K3's full technical report when this article was written, so the exact role of the "latent" expert representation should not be reverse-engineered from a launch diagram. What is confirmed is the 16-of-896 routing pattern and the training methods used to keep allocation stable at that scale.

Kimi Delta Attention changes the context equation

Standard softmax attention compares tokens with other tokens in the context. That direct access is expressive, but its computation grows quadratically with sequence length, and the key-value cache becomes a major inference cost as the context grows.

Kimi Delta Attention, or KDA, approaches the problem as a recurrent memory. It extends the gated delta rule with finer-grained control over what the memory retains, updates, and forgets. The goal is not to preserve every token in a full pairwise attention matrix. It is to maintain a compact state that can be updated efficiently as new tokens arrive.

Moonshot's earlier Kimi Linear paper combined KDA with occasional full-attention layers. In that 48-billion-parameter research model, the team reported up to a 75% reduction in key-value-cache use and as much as six times the decoding throughput at a one-million-token context. Those are results from Kimi Linear, not direct K3 benchmarks, but they explain why KDA is a useful foundation for a much larger long-context model.

Linear attention does not make context free, and a one-million-token input limit does not prove that a model can reliably use every relevant detail in a million tokens. Capacity, retrieval accuracy, latency, and output quality need separate evaluation. A production test should measure whether the model finds and applies the right evidence at different positions—not merely whether an API accepts the request.

Attention Residuals route information across depth

KDA deals with information flow across a sequence. Attention Residuals, or AttnRes, deals with information flow across model depth.

Standard residual connections add each layer's output to a running hidden state. This makes deep transformers trainable, but every contribution is accumulated with a fixed weight. As a network grows deeper, useful earlier signals can become diluted inside that uniform sum.

Attention Residuals replace that fixed accumulation with learned, input-dependent attention over earlier layer representations. A later layer can selectively retrieve the representations it needs instead of receiving the same unweighted accumulation every time.

The full method can be expensive because it retains outputs from many earlier layers. Moonshot's Block AttnRes groups layers into blocks and attends to block-level representations, reducing the memory and communication overhead while preserving most of the benefit.

This is easy to confuse with token attention, but the axes are different:

• KDA selects and compresses information across token positions.
• AttnRes selects information produced at different depths in the network.

Combining them gives the model control over both when information appeared in the sequence and where it was transformed in the model.

The architecture is a three-part scaling strategy

The most useful way to understand K3 is as three connected forms of selective computation:

1. Selective width: MoE activates a small subset of a very large expert pool.
2. Selective sequence memory: KDA decides what long-context information to retain and update.
3. Selective depth: AttnRes decides which earlier representations a later layer should reuse.

Moonshot attributes an approximately 2.5-times improvement in overall scaling efficiency over Kimi K2 to these architectural changes plus updated training and data methods. That figure is a vendor-reported scaling result, not an independent production benchmark. Still, it expresses the design goal well: convert additional compute into capability more efficiently instead of scaling a conventional transformer unchanged.

Why the benchmark headline needs context

The video's description places K3 near the strongest proprietary systems in benchmarks. Moonshot's own wording is more careful: it says K3 demonstrates frontier-level results while still trailing Claude Fable 5 and GPT 5.6 Sol in overall performance.

The detailed benchmark notes also show why a single ranking is hard to interpret. Models were evaluated with different agent harnesses, including Kimi Code, Claude Code, and Codex. Some tests used different hardware or fallback behavior. Reasoning effort, context management, and tool orchestration can materially affect the result of an agent benchmark.

This does not invalidate the scores. It means the result belongs to a complete system—model, harness, tools, prompts, context policy, and runtime—not to model weights alone.

What I would test before production use

For teams evaluating K3, a small benchmark suite built from real work will be more useful than repeating the public leaderboard. I would measure:

1. Completed-task cost, including retries and long reasoning traces, rather than price per token alone.
2. Long-session stability with full history, compacted history, and model switching.
3. Evidence retrieval at the beginning, middle, and end of very long contexts.
4. Tool-call reliability across multi-step coding or research tasks.
5. Latency and throughput under realistic concurrency and cache-hit rates.
6. Scope adherence when instructions are ambiguous or a tool can make consequential changes.

The last two tests are especially relevant because Moonshot documents several limitations. K3 can become unstable when a harness does not preserve its full thinking history, and its long-horizon training can make it overly proactive on underspecified tasks. The company also acknowledges a user-experience gap relative to the strongest proprietary models.

Those are production concerns, not footnotes. A powerful agent that improvises beyond the requested scope may require stricter system prompts, permissions, and approval boundaries than a less proactive model.

The practical takeaway

Kimi K3 matters less because it crossed a 2.8-trillion-parameter threshold than because it combines three kinds of routing in one system. It routes tokens to experts, sequence information into recurrent memory, and layer outputs across depth.

That architecture may allow very large models to increase capacity without paying the full dense-compute cost for every token. It also shifts complexity into routing stability, distributed serving, memory design, and the agent harness surrounding the model.

For production teams, the right question is therefore not "Is K3 larger?" It is "Does this complete system deliver better quality, reliability, latency, and cost on our work?" The architecture makes that outcome plausible. Only a workload-specific evaluation can establish it.

Sources

• Kimi K3 explained in 13 minutes, Caleb Writes Code
• Kimi K3: Open Frontier Intelligence, Moonshot AI
• Kimi Linear: An Expressive, Efficient Attention Architecture, Kimi Team
• Attention Residuals, Kimi Team