Optimizers & Stability

Adam held the frontier longer than any technique in this guide: a full decade from 2014. The first credible succession arrived in 2024 from a matrix-aware lineage, and stability engineering, once folklore, became the differentiating skill of trillion-scale training.

The optimizer lineage

Figure 1. Adam’s decade and the Muon succession. The pattern for a decade: challengers beat Adam at small scale and failed to displace it at frontier scale. Muon broke the pattern by proving itself at 1T parameters in an open release.

The schedule lineage

Schedule	Era	Status
Warmup + linear decay	2018 to 2020	Dead
Warmup + cosine to ~10%	2020 to 2023	The long default; requires knowing total steps upfront
WSD (warmup, stable, decay)	2024 onward	Current: flat plateau allows open-ended training, decay phase mines quality (annealing)
Batch-size ramping	2020 onward (GPT-3)	Standard companion at frontier scale

Stability, from folklore to checklist

Failure	Countermeasure	Established
Gradient blowups	Global-norm gradient clipping	Universal since GPT-3 (2020)
Loss spikes at scale	Skip-batch restarts (PaLM), then gradient-shape fixes	2022
Output-logit divergence	z-loss regularization	2022
Attention-logit blowups	QK-norm (proposed 2020); qk-clip (MuonClip)	At scale 2023 to 2025
BF16 gradient noise	Selective FP32 accumulations; see Precision	2022 onward
Late-training instability	WSD decay phase, EMA weights	2024

The measure of progress: stability went from accepted cost to competitive claim.

Choosing hyperparameters across scale

Technique	Introduced	Status
Per-model sweeps at full scale	to 2021	Dead at frontier cost
muP (maximal update parametrization): tune a small proxy, transfer the learning rate	2021 to 2022	Frontier practice (Cerebras-GPT, MiniCPM lineage); u-muP extends it to FP8
Fitted scaling rules for optimizer settings	2023 onward	Standard companion to scaling ladders

Ledger

Verdict	Techniques
Dead	Plain Adam without decoupled decay; cosine-only scheduling at frontier; spike surgery as standard practice
Current	AdamW (small and mid scale), Muon lineage (frontier open models), WSD schedules, qk-clip/z-loss stability stack, muP-style hyperparameter transfer
Contested	Per-Head Muon (vendor-stated, weights pending); second-order methods’ overhead-vs-win at frontier; optimizer choice interaction with MoE routing

Next: Precision & Parallelism, the numerics those updates run in.