Post-Training & Alignment

Post-training is where the most algorithm churn in the field happened: the reward-learning machinery changed three times in four years while the concept, shape behavior with preferences and rewards after pretraining, stayed fixed. It is also where product identity now lives; base models converge, post-training differentiates.

The lineage

Figure 1. The post-training lineage. Each stage kept its predecessors as pipeline components: a current frontier pipeline runs SFT, preference optimization, and RL phases in sequence, not as alternatives.

The algorithm churn

Generation	Mechanism	Why it was displaced
PPO-RLHF (2022)	Learned reward model + clipped policy gradients	Fragile, expensive, reward hacking; four models in memory
DPO family (2023 to 2024)	Closed-form preference loss on paired data	Offline; ceiling on eliciting new behavior
GRPO / RLVR (2024 to 2025)	Group-relative advantages, verifiable rewards, no value network	Not displaced; current, limited to checkable domains
Agentic RL (2025 to 2026)	Episode-level rewards over tool-use trajectories	Current frontier; harness-coupled (Agentic Patterns)

The machinery under GRPO

GRPO arrived in DeepSeekMath (Feb 2024) and became the default critic-free method. Its mechanism is narrow and worth stating exactly: generate a group of completions per prompt, score them, and use the group mean as the baseline. That removes the value network, which is where PPO spent a full model’s worth of memory.

Component	What it does	Source
Group-relative advantage	Group mean replaces a learned critic; no value network in memory	DeepSeekMath, arXiv:2402.03300
Large groups	Published configuration ran 16 prompts at group size 64, batch 1,024; large batches are a stability requirement, not a preference	DeepSeekMath
Rule-based reward only	R1 used accuracy (answer matching, compiler running test cases) plus format, and deliberately avoided neural reward models to prevent reward hacking at scale	DeepSeek-R1, arXiv:2501.12948
Reference implementation	GRPOTrainer in Hugging Face TRL, now with per-example environment selection and environment-owned rewards	TRL docs

The zero-variance consequence follows from the math and shapes practice: if every completion in a group scores identically, the advantage is zero and the gradient is zero. Tasks the model always solves and always fails both teach nothing.

DAPO (arXiv:2503.14476) named the fix and quantified the damage. As training proceeds the count of prompts at accuracy 1 keeps rising, so the effective prompt count per batch keeps falling, which raises gradient variance and dampens the training signal. Its dynamic sampling step over-samples and discards prompts at accuracy 0 and 1, refilling until every group in the batch has both correct and incorrect completions. Difficulty filtering is therefore structural to group-relative methods, not a tuning nicety.

Process rewards and their cost

Process supervision beat outcome supervision on hard reasoning in Let’s Verify Step by Step (OpenAI, May 2023): the process-supervised model solved 78% of a representative MATH test subset. The result came with a bill. It rested on PRM800K, 800,000 human step-level correctness labels, released alongside the paper.

That cost is the live tension in the contested row below. Process rewards work and are hard to scale; R1 went the other direction and used cheap checkable outcomes instead. The two results are not in conflict, they price differently.

What stayed constant

Constant	Detail
The data hierarchy	Demonstrations teach format, preferences teach taste, rewards teach correctness; every generation re-implements this split
Reward hacking	Every reward signal so far has been gamed (verbosity under RLHF, style bias under LLM judges, test-case exploitation under RLVR); countermeasures move, the phenomenon does not
The safety layer	Refusal training, system-prompt hierarchies, and constitution-style rules ride the same pipeline; deployed stacks add separate guard classifiers around the policy model (Llama Guard 2023 onward, constitutional classifiers 2025)
Calibration and abstention	Hallucination-reduction training entered the pipeline (2025 onward): abstention-aware rewards against the accuracy-only incentives that reward guessing
Model merging (2023 onward)	Quietly became standard operational practice between stages: average checkpoints from parallel runs rather than pick one

Ledger

Verdict	Techniques
Dead	PPO as the frontier default; pure-SFT products; human-label-only preference data at scale
Current	Multi-stage pipelines (SFT + preference + RLVR), GRPO-style critics-free RL, RLAIF, rejection sampling, difficulty filtering, asynchronous rollout collection, merging, guard-model stacks
Contested	Whether RL adds capability or elicits what pretraining stored; agentic-RL transfer across harnesses; process vs outcome rewards, where process wins on quality and loses on label cost

Next: Reasoning & Test-Time Compute, the capability RLVR was built to train.