Serving & Inference

Serving is where architecture meets the bill. Every technique in this chapter exists to raise tokens per second per dollar, and several chapters upstream (attention redesigns, MoE, quantization) were themselves driven by what this layer could not afford.

The lineage

Figure 1. The serving lineage. The unit of optimization moved from the request (static batching) to the token (continuous batching) to the memory page (PagedAttention) to the datacenter (disaggregation).

What each stage bought

Technique	Year	Gain
Continuous batching	2022	Order-of-magnitude throughput on mixed workloads
PagedAttention	2023	KV memory utilization to ~100%; bigger effective batches
Speculative decoding	2022 to 2023	2x to 3x decode speed at equal quality
Prefix caching	2024	Cache-hit input priced ~10x below cache-miss (Moonshot claims >90% hit rates for coding agents)
Multi-tenant LoRA serving	2023	Thousands of customer adapters on shared base weights (S-LoRA lineage); the mechanism behind fine-tune APIs
Disaggregated prefill/decode	2024	Independent scaling of compute-bound and memory-bound phases

Coupling with the rest of the stack

Upstream layer	Serving coupling
Attention	GQA/MLA/hybrid linear exist because KV cache dominates serving memory (Attention & Positions)
MoE	Low active fractions cut FLOPs per token but demand fleet-scale expert parallelism; trillion-parameter open models serve at ~64-accelerator granularity (K3 audit)
Reasoning	Thinking tokens are output tokens; serving economics now include the reasoning budget, and effort tiers are a serving feature as much as a capability one
Structured output	Constrained decoding (JSON modes, grammar masks) moved from library trick to API surface, because agent harnesses depend on parseable output

Ledger

Verdict	Techniques
Dead	Static batching; unmanaged KV allocation; FP16-only serving; per-request pricing blind to cache state
Current	Continuous batching + paged KV, speculative decoding, prefix caching, disaggregation, quantized serving, multi-tenant adapter serving, the vLLM/SGLang/TensorRT-LLM engine ecosystem
Contested	Speculative decoding under high load (acceptance-rate economics); linear-attention serving stacks (vLLM/SGLang support for KDA-class layers still landing)

Next: Evaluation, how any of these gains are measured at all.