Mixture of Experts

Mixture of experts is the oldest idea in this guide and the one with the longest dormancy. The 1991 formulation waited a quarter century for the hardware and routing tricks that made it practical, then became the frontier default in under three years. The trend inside the technique is a single number: the fraction of parameters active per token, falling steadily.

The lineage

Figure 1. The MoE lineage. The 2024 DeepSeek reformulation (many small experts, a shared always-on expert, loss-free balancing) is the base of every current trillion-scale open model.

The sparsity trend

Model	Year	Experts (active of total)	Active fraction of parameters
GLaM	2021	2 of 64	~8%
Mixtral 8x7B	2023	2 of 8	~27%
DeepSeek-V3	2024	8 of 256 + 1 shared	~5.5% (37B of 671B)
Kimi K2	2025	8 of 384 + 1 shared	~3.2% (32B of 1T)
Kimi K3 (vendor-stated)	2026	16 of 896	~1.8%

Total parameters grow faster than serving cost. GPT-4’s rumored MoE architecture (2023, never confirmed) marks where the closed frontier likely made the same move without disclosure.

What each generation solved

Problem	Failed or partial answer	Current answer
Experts collapse (all tokens to one expert)	Auxiliary load-balancing losses (2017 to 2023), which tax quality	Bias-based, loss-free balancing (DeepSeek-V3)
Expert granularity	Few large experts (Mixtral)	Many fine-grained experts + shared experts (DeepSeekMoE)
Communication cost	All-to-all overhead limits	Expert parallelism co-designed with the stack (Precision & Parallelism)
Routing space	Token-choice softmax over experts	Still token-choice; latent-space routing is K3’s vendor-stated bet

Ledger

Verdict	Techniques
Dead	Dense frontier monoliths (Llama 3.1 405B as the last landmark); top-1 Switch routing; heavy auxiliary balancing losses
Current	Fine-grained + shared experts, loss-free balancing, top-k token choice, expert parallelism
Contested	Latent routing (K3, uninspectable until weights ship); expert-choice variants; upcycling dense checkpoints into MoE

Next: Tokenization, the interface every parameter sees.