Model

Attention & Positions

From full multi-head attention to hybrid linear: the KV-cost lineage, and how positional encoding converged on RoPE.

Attention is the most redesigned component in the stack, and every redesign has the same motive: the KV cache. Full multi-head attention stores keys and values for every head at every position; at long context that memory dominates serving cost. Positions followed the opposite path: a crowded field of encodings collapsed onto one survivor.

The attention lineage

Figure 1. The KV-cost lineage. Each step cuts cache size per token. The frontier forked in 2025: hybrid linear (Kimi lineage) versus trained sparsity (DeepSeek lineage), two competing answers to the same cost.

Technique	Year	KV cache relative to MHA	Status
Multi-head attention	2017	1x	Small models only
Multi-query attention	2019	1/heads	Superseded by GQA (quality loss)
Grouped-query attention	2023	1/group-size	The open-model default
Multi-head latent attention	2024	~1/10 and below	DeepSeek lineage; K2
Sliding window	2020 to 2023	bounded	Alive as an interleaving component
Hybrid linear + full	2025 to 2026	near-constant in length for linear layers	Frontier long-context bet (K3 audit)

FlashAttention (2022, v2 2023, v3 2024) sits outside the lineage: it changed the cost of computing exact attention, not the definition, and in doing so killed most of the 2020-era approximate-attention zoo before those methods reached the frontier.

The positions lineage

Figure 2. Positions converged. RoPE won on quality, simplicity, and how well it takes post-hoc extension methods. Extension methods continue in Long Context.

The block lineage

The transformer block’s other components ran the same convergence as positions:

Figure 3. The quiet survivors. RMSNorm and SwiGLU appear in every open architecture this guide cites; QK-norm (Optimizers & Stability) extends the same normalization lineage into attention itself.

Ledger

Verdict	Techniques
Dead	Learned absolute positions; fixed sparse-attention patterns (Longformer/BigBird global tokens); pure linear attention as a full replacement
Current	GQA (open default), MLA and trained sparse attention (DeepSeek lineage), RoPE (near-universal), FlashAttention kernels
Contested	Hybrid linear (Kimi lineage) vs trained sparse (DeepSeek DSA) as the long-context answer; hybrid ratios (3:1 in Kimi Linear; K3’s undisclosed); SSM hybrids (Jamba lineage)

Next: Mixture of Experts, the other half of the frontier’s cost equation.