Vision

Vision is the modality with the deepest capability stack. Multimodality covers how vision got into the language model; this chapter covers what it can do once inside, and the encoder and resolution lineages that gate every capability on the ladder.

The encoder lineage

Figure 1. The encoder lineage. Resolution, not architecture, became the differentiator: fixed square crops lost to packing and tiling because documents and screens are not squares.

The capability ladder

Figure 2. Each rung reuses the one below: grounding needs reading, computer use needs grounding, agents need all of it.

Capability	Turning point	Current state
OCR and documents	OCR-free models (2021 to 2022) ended the separate-OCR-stage assumption	Frontier VLMs approaching dedicated-pipeline accuracy but still trailing specialist OCR systems on document benchmarks; vendor document scores are launch-table staples (K3 claims OmniDocBench 91.1, vendor-stated)
Charts and tables	Chart reasoning entered standard eval suites (2023 to 2024)	Reliable on clean renders; degrades on dense or low-quality scans
Grounding	Coordinate output in open VLMs (2023)	Boxes and points standard in open models; closed frontier exposes it selectively
Video	Long context made frames affordable (2024)	Hour-scale understanding shipping; fine-grained temporal reasoning still weak
Computer use	Screenshot-to-action agents (2023 to 2024)	Shipping in agent products; reliability, not perception, is the bottleneck

What gates the ladder

Gate	Why it binds
Resolution vs token budget	Every image is a token bill; dense documents need thousands of tokens or lose small text (Serving)
Spatial precision	Language-first decoders localize approximately; grounding accuracy trails detection-specialist models
Temporal modeling	Frame sampling into context is not motion understanding; video evals reward retrieval over dynamics
Hallucination	Vision inherits the text model’s assertiveness; models describe what plausibly should be in an image (Reasoning)

Ledger

Verdict	Techniques
Dead	Separate OCR stages feeding text-only models at the frontier; fixed low-resolution square inputs; task-specific vision heads per capability
Current	Contrastive ViT towers (SigLIP lineage), dynamic resolution and tiling, coordinate-grounded output, screenshot-driven computer use, video via long context
Contested	Encoder-free architectures at frontier scale; whether GUI agents need vision-specific training or general agentic RL suffices; specialist parity for detection and dense OCR

Next: Long-Horizon Agents, where every capability in this group gets spent.