Training Data

Data went from an afterthought to the binding constraint in three years, then hit legal and supply walls and pivoted to sources that models and environments generate themselves. Quality filtering, not raw volume, is the technique that carried the era.

The lineage

Figure 1. The substrate lineage. Every stage kept the previous one as a layer: the modern mix is filtered web + licensed corpora + synthetic + interaction data, staged across training.

What the pipeline learned

Finding	Year established	Consequence
Deduplication improves loss and reduces memorization	2021 to 2022	Exact and fuzzy dedup became mandatory
Web-only can match curated mixes when filtered well	2023 (RefinedWeb)	Killed the curated-mix orthodoxy
Model-based quality classifiers beat heuristics	2024 (FineWeb-Edu, DCLM)	Filtering became a modeling problem
Data can be reused ~4 epochs before returns collapse	2023 (data-constrained scaling)	Softened the “data wall” deadline
High-quality data belongs at the end of training	2023 to 2024 (annealing/midtraining)	Data became staged, not shuffled

Staging, the current shape

Stage	Data
Pretraining	Filtered web at maximum volume, multilingual and code-weighted
Midtraining	Long-context extension mixes, domain injections, higher-quality re-weighting
Annealing	Small, high-quality, benchmark-adjacent corpora as the learning rate decays; a known contamination risk (Evaluation)
Post-training	Instruction, preference, and verifier-filtered reasoning data; increasingly synthetic (Post-Training)

The walls and the pivot

• Licensing became a market (Reddit, news publishers) while litigation set boundaries case by case; robots.txt closures shrank the refreshable open web.
• Synthetic generation moved from augmentation trick to primary source for post-training; model collapse concerns were bounded to closed-loop regimes with no fresh signal.
• Interaction data is the current chapter: agentic trajectories and environment rollouts are proprietary, self-renewing, and the subject of provenance disputes at frontier labs (the February 2026 report on harvested exchanges; see our K3 audit).

Ledger

Verdict	Techniques
Dead	Raw undeduplicated dumps; shadow-library corpora; “more epochs freely” assumptions; hand-curated mixes as the sole strategy
Current	Classifier-filtered web, staged data schedules, licensed corpora, verifier-filtered synthetic data
Contested	Synthetic majority in pretraining; interaction-data provenance and consent; how far annealing can go before it is benchmark training

Next: Objectives & Pretraining, what the model is asked to do with the data.