Precision & Parallelism

Two lineages define the systems layer. Precision moved one direction only, 32 bits to 4, each halving buying tokens per dollar. Parallelism accreted: no strategy ever replaced another, and a frontier run now composes five of them at once.

The precision lineage

Figure 1. The training-numerics lineage. FP8 required per-block scaling and selective FP32 accumulation; 4-bit QAT moves quantization from a post-hoc compression step into training itself (Distillation & Compression covers post-hoc quantization).

Precision	First frontier use	What it required
FP16	2018	Loss scaling to dodge underflow
BF16	~2019 (TPU stacks); GPU default from 2021	Ampere/TPU hardware; removed loss scaling
FP8	2023 vendor-claimed; 2024 open (DeepSeek-V3)	Transformer Engine scaling, FP32 master weights and accumulations
MXFP4	2025 to 2026	Microscaling block formats, QAT from the post-training/SFT stage

The parallelism stack

Figure 2. Parallelism accretes. Each axis solves one memory or compute wall; frontier training composes all of them, and model architecture is now co-designed with the parallelism plan rather than handed to it.

Kernel-level systems follow the same one-way pattern as precision:

Kernel technique	Status
Activation checkpointing	Standard by 2020
FlashAttention	2022 onward; made exact attention IO-bound rather than compute-bound
Fused kernels	Absorbed what were once separate ops

Ledger

Verdict	Techniques
Dead	FP32 training; FP16-with-loss-scaling at frontier; unsharded data parallelism beyond small scale
Current	BF16 baseline, FP8 at the efficiency frontier, 3D+EP+CP composition, FSDP as the open-stack default, FlashAttention kernels
Contested	MXFP4 quality parity (vendor-stated pending K3 weights); how far below 4 bits training can go (ternary/BitNet research); cross-datacenter low-communication training (DiLoCo lineage, INTELLECT runs)

Next: Scaling Laws, the discipline that decides how to spend what this stack provides.