Long-Horizon Agents

Long-horizon capability is the axis the current era competes on: not answering, but completing multi-step work. The model-side lineage below runs from prompted loops that failed by compounding error to reinforcement learning over full task episodes. The harness layer that carries these models in production has its own guide; this chapter stays on the model side of that boundary.

The lineage

Figure 1. Agency moved from prompt to parameter. The 2023 autonomous-loop wave had the right ambition and no training signal; agentic RL supplied it.

Milestones

Milestone	Date	What it established
ReAct	2022	Reason-act interleaving as a prompting pattern
Toolformer	Feb 2023	Self-supervised tool-call learning
AutoGPT wave	2023	Demand proven; prompted loops shown insufficient
OpenAI function calling	Jun 2023	Tools as structured API surface
SWE-bench	2023	Real repository work as the benchmark
Devin, SWE-agent	2024	Agent-computer interfaces for coding
Claude computer use	Oct 2024	General GUI operation at the frontier
MCP	Nov 2024	An open protocol for the tool boundary
Deep-research agents	Dec 2024 to 2025	Minutes-long autonomous browsing and synthesis as product
Agentic RL at open frontier	2025 to 2026	K2 positioning, R1-lineage tool RL, K3 Swarm (vendor-stated)

The horizon as a metric

• The task horizon replaced single-response accuracy as the frontier measure: METR’s 2025 study measured the task length agents complete at 50% reliability and found it doubling roughly every 7 months.
• Cost per completed task joined accuracy as a first-class evaluation axis (Evaluation).
• Agentic benchmarks (SWE-bench Verified, tau-bench, Terminal-Bench, GAIA lineage) are harness-sensitive: the same weights score differently across scaffolds (K3 audit).

How long-horizon training actually works

The chapter above tracks what agents can do. This section covers how the training runs, because the horizon changes the method rather than just extending it.

The governing constraint is credit assignment. A long episode produces roughly one bit of reward after hundreds of interdependent decisions, and that reward cannot be densified without corrupting it. Any cheap intermediate proxy gets gamed in proportion to the number of steps optimized against it.

Technique	Mechanism	Evidence
Verifiable outcome reward	Program checks the end state: tests pass, code compiles, answer matches. Hard to fake because faking requires doing the task	R1’s compiler-and-test-case rewards, arXiv:2501.12948
Group rollouts	Run the same task N times, score against the group mean; supplies a baseline without a critic	DeepSeekMath, arXiv:2402.03300; TRL GRPOTrainer
Dynamic sampling	Over-sample and discard prompts at accuracy 0 and 1, refilling until every group holds both correct and incorrect completions; without it the effective prompt count per batch decays as training proceeds	DAPO, arXiv:2503.14476
Rejection sampling / expert iteration	Generate many trajectories, keep the successes, fine-tune on those. Cheaper and more stable than full RL	SWE-Gym, arXiv:2412.21139
Trained verifiers	Outcome-supervised reward models trained on successful and failed trajectories, used to re-rank at inference	SWE-Gym
Asynchronous rollouts	Decouple generation from learning so minutes-long episodes do not stall the optimizer; tolerate bounded staleness	Asynchronous RLHF, arXiv:2410.18252, ICLR 2025
Trained compaction	Optimize task execution and summary generation jointly, so compressing one’s own history becomes a learned skill rather than a blocking side call	CompactionRL, arXiv:2607.05378
Executable environments	Real containers, real repositories, real dependency trees; the scarce input is verifiable environments, not data	SWE-Gym: 2,438 unit-test-validated instances from real GitHub issues

Two measured results anchor the value. SWE-Gym fine-tuned a 32B Qwen-2.5 coder model on 491 sampled trajectories for +12.3% on SWE-Bench Lite and +13.6% on SWE-Bench Verified. Asynchronous RLHF trained ~40% faster on instruction-following and ~70% faster on GSM8k while matching synchronous accuracy, and found online DPO the most robust to off-policy data, with robustness rising as the policy model scales.

Why length is a different capability class

Some behaviors have no gradient signal at all until the horizon exceeds the payoff delay. Reading a codebase before editing, writing a test that catches a later bug, and checking an assumption all carry negative immediate value: they burn tokens and produce nothing. Short-episode training actively penalizes them. Only a reward landing hundreds of steps later registers the orientation cost as the thing that made the difference.

Error recovery has the sharper version of this property. It cannot be hand-authored as data, because the failures that matter are the ones the current policy actually makes. Long-horizon training is the only regime that generates that curriculum against itself, which is why SWE-Gym’s failed trajectories are training signal rather than waste.

What limits the horizon

Limit	Mechanism	Model-side response
Compounding error	Small per-step failure rates multiply across steps	Trained self-correction and recovery inside RL episodes
Context exhaustion	Long sessions overflow even 1M windows, and quality degrades before the limit as attention spreads over low-signal history	Trained compaction (CompactionRL optimizes summarizing jointly with the task); preserved-reasoning contracts (Long Context)
Reward hacking at horizon	Long episodes multiply gaming opportunities	Verifiable environment rewards; process auditing (Post-Training)
Confident error propagation	One hallucinated assertion feeds forty tool calls	Unresolved; hallucination rates rose with capability in measured cases (K3 audit)
Harness dependence	Same weights, different scaffold, different horizon	Official harness support and trained context contracts; the boundary is the Agentic Patterns layer

Ledger

Verdict	Techniques
Dead	Prompted autonomous loops as products (AutoGPT pattern); single-shot function calling marketed as agency; agent capability claims without task benchmarks
Current	Agentic RL over tool episodes, verifiable environment rewards, group rollouts with dynamic sampling, rejection-sampling expert iteration, trajectory-trained verifiers, asynchronous rollout collection, trained context compaction, interleaved reasoning and tool use, computer use, MCP-standardized tools, subagent orchestration, horizon and cost-per-task metrics
Contested	Whether the METR doubling trend holds; swarm-scale orchestration value (K3 Swarm is vendor-stated); whether agentic skill transfers across harnesses or binds to the training scaffold; whether long-horizon training teaches transferable agency or environment-specific priors

Next: Distillation & Compression, how capabilities move between models.