Training

Scaling Laws

Kaplan, Chinchilla, overtraining, data-constrained and test-time regimes: how the field prices compute before spending it.

Scaling laws are the field’s pricing discipline: fit a power law on small runs, extrapolate, then commit the budget. The discipline survived every era; its constants did not. Each regime change below rewrote where a compute budget should go.

The lineage

Figure 1. Three regime changes in six years. Kaplan priced parameters, Chinchilla repriced data, overtraining priced inference, and test-time compute added a second axis that training laws do not govern.

What each regime got wrong

Regime	Planning rule	What broke it
Kaplan (2020 to 2022)	Spend on parameters	Undertrained giants (Gopher); data was mispriced
Chinchilla (2022 to 2023)	~20 tokens per parameter	Optimizes training FLOPs only; serving cost ignored
Overtraining (2023 onward)	Overshoot data for small deployable models	Data supply and reuse limits
Test-time (2024 onward)	Buy accuracy with thinking tokens	Only prices verifiable domains; costs recur per query (Reasoning)

Companion debates

Debate	The dispute	Where practice landed
Emergence	Abrupt capability jumps (2022) versus metric artifacts of discontinuous scoring (2023)	Plan with smooth proxy metrics, expect surprises anyway
Extrapolation risk	Laws fit at small scale have missed regime boundaries (long-context behavior, RL post-training effects)	Staged commitment: scaling ladders with go/no-go gates rather than single extrapolations
Specialized laws	MoE sparsity, precision, vocabulary size, and distillation each acquired their own fitted laws from 2023 to 2025	”A scaling law for X” became the field’s standard paper template

Ledger

Verdict	Techniques
Dead	Kaplan constants as planning targets; single-law extrapolation across regimes; “params are all that matters”
Current	Chinchilla-style joint fits as baseline, inference-adjusted overtraining, data-constrained corrections, scaling ladders with gates
Contested	Test-time compute’s ceiling in unverifiable domains; whether agentic capability scales by any measurable law yet

Next: Long Context, the capability whose costs the laws above kept mispricing.