Objectives & Pretraining

The pretraining objective is the strongest example of frontier concentration: a crowded 2019 field of objectives collapsed to one winner by 2022, and innovation moved to what surrounds the objective rather than the objective itself.

The objective lineage

Figure 1. The objective converged, then accreted. FIM and MTP are additions on top of next-token prediction, not replacements; MTP’s main payoff is speculative decoding at serving time (Serving).

Why next-token won:

• Every token supplies signal (masked objectives train on ~15% of positions).
• The objective is the product interface: generation needs no fine-tuning bridge.
• It composes with everything downstream: in-context learning, instruction tuning, and RL all operate on the same sampling loop.

The staging lineage

Figure 2. Pretraining became a schedule of phases. The staging absorbed jobs that once had separate names: long-context training, domain adaptation, and part of what distillation used to do.

Stage	Typical content	Introduced broadly
Pretrain	Filtered web + code, max volume	2019
Midtrain	Long-context mixes, domain data, synthetic infusions	2023 to 2024
Anneal	Small high-quality sets under decaying LR	2023 to 2024
Continual	New modalities or data on an existing checkpoint	2024 to 2026

Ledger

Verdict	Techniques
Dead	Masked LM for generation; span corruption at frontier; objective mixtures (UL2); from-scratch restarts per model version
Current	Next-token prediction, FIM for code models, staged curricula, continual pretraining
Contested	Multi-token prediction as quality (vs serving) win; how much reasoning belongs in pretraining vs post-training; diffusion LMs as a serving-speed challenger to autoregression

Next: Optimizers & Stability, the machinery that makes the objective trainable at scale.