Long Context

Context length grew a thousandfold in seven years, and the technique that carried most of it is unglamorous: stretch RoPE, then train briefly at the new length. The open problem moved from reaching a context length to being reliable across one.

The lineage

Figure 1. The context lineage. Exact attention with better kernels plus RoPE stretching won, and the million-token tier is now an architecture question again.

Milestone	Year	Standard window
GPT-3	2020	2K
GPT-4 launch tiers	2023	8K to 32K
GPT-4 Turbo / Claude 2.1 era	2023	128K to 200K
Gemini 1.5	2024	1M (10M in research)
Open-weight frontier (K2.5, K3)	2026	256K to 1M

Claimed vs usable

Gap	Detail
Needle tests saturate first	Single-fact retrieval (“needle in a haystack”) passed long before multi-fact reasoning did; RULER-style aggregate evals and long-conversation tasks still show degradation well inside claimed windows
Lost in the middle (2023)	Retrieval quality dips for mid-context positions; mitigated, not eliminated, in current models
Training length lags claimed length	Extension methods extrapolate; reliability tracks the longest well-trained length, which labs rarely disclose
Cost gates usage	Quadratic prefill and linear KV growth mean the harness layer still compacts context in practice (Agentic Patterns: Memory); K3’s launch benchmarks report a compaction-on variant scoring above the full-window run (K3 audit)
Beyond the window	Corpora larger than any window still belong to retrieval: RAG remains the production pattern at corpus scale, with retrieval-in-pretraining (RETRO, 2021) the dead branch
Eviction artifacts	Attention-sink tokens (2023) explain why streaming eviction works; shipped as trained sink parameters in open weights

Ledger

Verdict	Techniques
Dead	Fixed 2K horizons; the standalone sparse-attention zoo; retrieval as the only long-document strategy
Current	RoPE scaling (YaRN lineage) + long-context midtraining, ring/context parallelism, hybrid linear serving, harness-level compaction, retrieval as the complement at corpus scale
Contested	10M-class claims (thinly evidenced); whether full-window reliability is achievable or compaction remains permanent

Next: Post-Training & Alignment, where capability stops being about the base model.