The LLM Timeline: What Lived and What Died

Capability emerged from scale alone. The jump from GPT-2 to GPT-3, roughly 100x in parameters with no architectural change, produced in-context learning: the model performed new tasks from examples in the prompt, a behavior nobody trained for. Kaplan's scaling laws made loss a predictable function of size, and parameter count became the industry's planning variable. The field consolidated onto one architecture, the decoder-only transformer, and has not left it since.

The field learned it had been spending compute wrong. Gopher at 280B showed diminishing returns; Chinchilla at 70B, trained on four times the tokens, beat it across the board. The constraint moved from parameters to data, and token budgets started appearing next to parameter counts in every paper. PaLM at 540B closed the era as the largest dense model ever disclosed, a record that still stands because the strategy it represented was abandoned.

Post-training turned base models into products. Instruction tuning plus RLHF converted raw next-token predictors into assistants that follow directions, and ChatGPT took that to a hundred million users in two months. The frontier went closed: GPT-4 shipped without an architecture disclosure, ending the parameter-count era of marketing. In the same window LLaMA's weights leaked, and an open ecosystem formed around fine-tuning at consumer scale.

The trend inverted: hold capability constant and cut the cost of serving it. GQA shrank the KV cache, Mixtral made mixture-of-experts open and practical, quantization moved inference to consumer hardware, and small models overtrained far past Chinchilla-optimal (Mistral 7B) matched the previous generation's giants. Context windows stretched from 4K to a million tokens, and the frontier went natively multimodal. The economics of inference, not training, drove design.

A second scaling axis opened at inference time. o1 trained chain-of-thought with reinforcement learning on verifiable rewards, making accuracy something you buy with thinking tokens rather than parameters. DeepSeek-R1 reproduced the recipe in the open within four months at disclosed cost, and distillation pushed reasoning down into small models almost immediately. Math and code benchmarks fell fast, because verifiable domains are exactly where the recipe works best.

The unit of value shifted from an answer to a completed task. Reinforcement learning moved to long-horizon agentic work, and coding harnesses became the benchmark that matters. Open weights reached trillion-parameter scale through extreme sparsity (K2's 32B active of 1T, K3's roughly 50B of 2.8T), trained natively in 4-bit precision, with hybrid linear attention making million-token contexts serveable. The open question this era will answer: whether harness-dependent agentic skill transfers the way raw capability did.