Distillation & Compression

Distillation is 2015’s technique having its best decade: every era found a new thing worth transferring, and by 2026 the practice is economically central and legally contested. Compression’s other track, quantization, ran a parallel path from post-hoc trick to native training format.

The distillation lineage

Figure 1. What gets transferred moved up the stack: labels, tasks, instructions, logits, reasoning. Each step made students stronger and provenance murkier; the current chapter is legal and forensic as much as technical.

The quantization lineage

Figure 2. Quantization moved upstream. The 2023 ecosystem quantized other people’s checkpoints; the 2026 frontier trains in the deployment format, deleting the quality gap by construction (Precision).

Compression track	Peak technique	Status
Distillation	Reasoning-trace distillation	Current; provenance-contested
Weight quantization	4-bit QAT / native MXFP4	Current
Pruning / sparsity	SparseGPT, Wanda (2023)	Research-alive, deployment-marginal
Low-rank adaptation	LoRA (2021), QLoRA (2023)	Current for customization, not frontier training

The provenance dispute

• Closed labs ban training on their outputs by terms of service; enforcement moved from policy to detection (fingerprinting, forensic analysis of derivative models).
• The February 2026 report alleging 16M+ harvested frontier-model exchanges across three labs (3.4M+ attributed to Moonshot) made the dispute concrete at frontier scale; no public evidence ties those datasets to a specific release (K3 audit).
• Open-weight licenses meanwhile normalized distillation of open models: R1’s license explicitly permits it, and distilled variants ship officially.

Ledger

Verdict	Techniques
Dead	Post-hoc INT8 as the deployment ceiling; distillation as merely a small-model trick; hand-tuned per-layer bit allocation
Current	Reasoning distillation, on-policy logit distillation, 4-bit QAT and native formats, LoRA-family adaptation
Contested	Output-distillation legality across labs; detection reliability; dynamics-level transfer (our research direction)

Next: Serving & Inference, where compression’s savings are realized.