LLM Development Techniques

This guide maps the techniques behind large language models as lineages. Each chapter takes one domain, draws the succession of techniques from first use to current frontier practice, and records what died, what survived, and what replaced what. The companion timeline post covers the same history chronologically by model release; this guide covers it by domain.

How to read a chapter

• Every chapter centers on one or more lineage diagrams: vertical flowcharts where each node is a technique, dated by first prominent use, and each edge means “replaced or absorbed”.
• Tables carry the exact facts: technique, year, origin, status.
• A status ledger closes most chapters: which techniques are dead, which are current, which are contested.
• Claims about undisclosed frontier systems are marked vendor-stated or reported. Dates are first-publication or first-prominent-deployment at month or year granularity.

The domains

Figure 1. Seventeen chapters in four groups.

Group	Chapter	The lineage it draws
Model	Attention & Positions	MHA to hybrid linear; positions to RoPE
Model	Mixture of Experts	Dense to 1.8%-active sparsity
Model	Tokenization	BPE’s reign and the byte-level challengers
Training	Training Data	Curated scrape to synthetic and interaction data
Training	Objectives & Pretraining	Why next-token won; the staging that grew around it
Training	Optimizers & Stability	Adam’s decade to the Muon lineage
Training	Precision & Parallelism	FP32 to MXFP4; the parallelism stack
Training	Scaling Laws	Kaplan to test-time-compute regimes
Capability	Long Context	1K to 1M tokens, claimed vs usable
Capability	Post-Training & Alignment	SFT to agentic RL
Capability	Reasoning & Test-Time Compute	Prompt trick to trained behavior
Capability	Multimodality	Vision and audio moving inside the language model
Capability	Vision	The capability ladder from OCR to computer use
Capability	Long-Horizon Agents	Prompted loops to trained agency and the task horizon
Deployment	Distillation & Compression	Knowledge transfer and its disputes
Deployment	Serving & Inference	The economics of tokens out
Deployment	Evaluation	GLUE to agentic harnesses

The recurring pattern

Three regularities appear in almost every domain, and the chapters return to them.

Pattern	Evidence across chapters
Techniques die by absorption, not refutation	Chain-of-thought prompting became trained reasoning; sparse-attention ideas resurfaced inside hybrid linear attention; RLHF’s reward-model concept survived three algorithm changes
The constraint migrates	Parameters, then data, then inference cost, then verifiable reward signal, then task horizon; each migration reset a domain’s frontier technique
Frontier practice concentrates	Domains with many competing techniques in 2020 (positions, attention patterns, objectives) converged to one or two survivors by 2024; the variation moved to a new layer

Deliberate exclusions: world models and JEPA-class architectures (the competing bet, not LLM development; our own Knowledge as Dynamics works in that space), robotics VLA systems (robot-policy development consuming VLMs), and federated training (frontier practice is centralized).

Related Veso work: the Agentic Patterns guide covers the harness layer above the model; the model matrix tracks current standings; Knowledge as Dynamics is our research contribution to the distillation domain.