Multimodality

Multimodality entered the stack from outside: separate vision models bolted onto frozen language models through progressively thinner bridges, until the bridge disappeared and the modalities moved into pretraining itself. The generation side ran a separate track (diffusion models outside the LLM) that is now converging on the same endpoint.

The understanding lineage

Figure 1. The bridge got thinner until it vanished.

The generation track

Figure 2. Generation converged from the other direction. Diffusion stayed dominant for standalone image quality; frontier assistants moved generation inside the model for instruction-following and editing fidelity.

Milestones

Milestone	Date	What it established
CLIP (OpenAI)	Jan 2021	Web-scale contrastive pairing; the vision encoder every adapter era reused
Neural audio codecs (SoundStream, EnCodec)	2021 to 2022	Discrete audio tokens; the enabling step for end-to-end speech models
Whisper (OpenAI)	Sep 2022	Web-scale supervised ASR; robust speech recognition as a commodity component
Flamingo (DeepMind)	Apr 2022	Frozen-LM cross-attention; few-shot multimodal
BLIP-2 / LLaVA	2023	Cheap bridges: Q-Former and a linear projection; visual instruction tuning
GPT-4 vision	2023	Frontier vision under a closed architecture
Gemini 1.0	Dec 2023	Stated natively multimodal from the start of pretraining
GPT-4o	May 2024	Single model, end-to-end text, vision, audio; realtime voice latency
Chameleon (Meta)	May 2024	Early-fusion discrete tokens for understanding and generation
Text-to-video (Sora, Veo lineage)	2024	The generation track’s video milestone; carries the world-model framing
Gemini 1.5	2024	Hour-scale video in a 1M context (Long Context)
Native image generation in assistants	2025	Instruction-faithful editing beats prompt-to-diffusion pipelines
K2.5 / K3 vision	2026	Open-weight vision: K2.5 via continual pretraining on an existing checkpoint; K3 native, weights due late July 2026 (K3 audit)

What settled and what did not

Front	State
Vision input	Settled on ViT-lineage encoders feeding the decoder, with dynamic resolution and tiling replacing fixed-square crops (2024 onward); open whether the separate encoder survives at all or early fusion absorbs it
Visual representation	The live fork: continuous embeddings (most deployed models) versus discrete tokens (Chameleon lineage, generation-friendly); neither has won
Audio	Followed vision’s path faster: cascaded ASR-LLM-TTS pipelines gave way to end-to-end speech in omni models; cascades survive where controllability and cost matter
Modality retrofits	The open-weight path: add a modality to an existing checkpoint, either by adapter training around a frozen model (Llama 3.2) or by full continual pretraining (K2.5), rather than retraining from scratch (Objectives & Pretraining)
Evaluation	Lags: multimodal benchmarks (MMMU and successors) sit earlier on the saturation-contamination curve than text (Evaluation)

Ledger

Verdict	Techniques
Dead	Caption-only vision-language training; fixed low-resolution square inputs at the frontier; cascaded pipelines as the frontier voice interface
Current	Native multimodal pretraining, ViT encoders with dynamic resolution, visual instruction tuning, continual-pretrain modality retrofits, end-to-end audio in omni models
Contested	Discrete vs continuous visual tokens; encoder-free early fusion at frontier scale; diffusion vs in-model generation for standalone image quality

Next: Vision, what the dominant modality can do once inside.