Benchmarks, cost curves, and the data behind the agentic AI transition.
SWE-Bench: Agents Solving Real GitHub Issues
SWE-Bench presents real GitHub issues and evaluates if the model's code fix passes the actual test suite. No simulations.
Jul '23
—
1.96% · GPT-4 (no scaffolding)
Jan '24
13.9%
13.9% · SWE-agent + GPT-4
May '24
22.0%
22.0% · Devin (Cognition)
Aug '24
41.0%
41.0% · Claude 3.5 Sonnet + scaffolding
Oct '24
49.0%
49.0% · Claude 3.5 Sonnet (Oct release)
Early '25
50%+
50%+ · Multiple frontier systems
📊 Key Insight
From 2% to 49% in 14 months is not a linear improvement — it is a phase transition. The same trajectory that moved SWE-Bench from 2% to 49% is now beginning on harder benchmarks (WebArena, OSWorld). Engineers who understand the architecture behind these numbers are 18 months ahead of the curve.
Success rate, steps to completion, and true cost for complex multi-step agent tasks. Hover bars for details.
⚠️ The Cheap Model Trap
GPT-4o-mini costs ~4× less per token than GPT-4o. But on complex agentic tasks, it fails 49% of the time (vs ~28% for GPT-4o) and averages 6.1 steps when it doesn't fail (vs 4.2). Total cost per completed task is higher with the cheaper model — before accounting for cost of failure in production.
78%
Claude 3.5 Sonnet complex task success
4×
More retries with cheap models
3.8
Avg steps (Claude) vs 6.1 (mini)
Sources: Berkeley Function Calling Leaderboard · Internal benchmarks · HELM
Cost vs Performance: The Frontier Model Case
True cost per completed task (including failures and retries) plotted against task success rate. Hover points for model details.
💡 The Counter-Intuitive Finding
The models that appear cheapest at the token level are often the most expensive at the task level. A model with 50% success that costs $0.04/task actually costs $0.08/task when you account for retries — before adding the cost of failed tasks that require human remediation (typically $5-50 in engineering time). The frontier model that costs $0.20/task but succeeds 78% of the time is the better economic choice for most production agents.