Model

Tokenization

BPE's long reign, the vocabulary-size race, and the byte-level challengers that have not yet taken the frontier.

Tokenization is the stack’s most stable layer. The algorithm GPT-2 shipped in 2019, byte-level BPE, is still what frontier models use in 2026. What changed is the vocabulary size, the training corpus behind it, and a growing research case that the layer should not exist at all.

The lineage

Figure 1. One algorithm, growing vocabularies. The byte-level challengers attack the layer’s existence rather than its parameters, and none has shipped in a frontier model yet.

The vocabulary race

Tokenizer generationYearsVocab sizeNote
GPT-2 / GPT-3 BPE2019 to 2020~50KEnglish-heavy merges
Llama 1/2 SentencePiece202332KSmall vocab, high fertility on non-English
cl100k (GPT-4)2023~100KCode-aware
Llama 32024128KSwitched to tiktoken-style BPE
o200k (GPT-4o)2024~200KMultilingual rebalance
Gemma / Gemini lineage2024 to 2026256KThe current upper end

Larger vocabularies buy compression (fewer tokens per document, so more effective context and cheaper serving) at the cost of embedding-table parameters and rare-token undertraining.

Persistent failure modes

Failure modeWhat happens
Fertility inequalityLanguages distant from the merge corpus pay 2x to 4x more tokens for the same content, silently pricing out non-English users per character of output
Numeric splittingInconsistent digit tokenization degrades arithmetic; modern tokenizers force single-digit or fixed-width splits
Glitch tokensUndertrained vocabulary entries produce anomalous behavior; documented across GPT-2 through Llama lineages
Boundary artifactsTrailing-whitespace and prefix-space asymmetries remain a source of subtle prompt bugs in every BPE model

Ledger

VerdictTechniques
DeadWord-level vocabularies; character-level modeling at scale; 32K vocabularies at the frontier
CurrentByte-fallback BPE at 100K to 256K, digit-aware splitting, multilingual-weighted merge corpora
ContestedByte-latent / patch-based models (research-strong, unshipped); “tokenization causes X” claims pending byte-level frontier evidence

Next: Training Data, the corpus those tokens come from.