WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation
:::info Stub — Full Engineering Breakdown Coming This paper has a linked code implementation and was featured on Hugging Face Papers with 41 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Shuangrui Ding et al. |
| Year | 2026 |
| HF Upvotes | 41 |
| arXiv | 2605.10912 |
| Download | |
| Code | https://github.com/internlm/WildClawBench |
Abstract
Large language and vision-language models increasingly power agents that act on a user's behalf through command-line interface (CLI) harnesses. However, most agent benchmarks still rely on synthetic sandboxes, short-horizon tasks, mock-service APIs, and final-answer checks, leaving open whether agents can complete realistic long-horizon work in the runtimes where they are deployed. This work presents WildClawBench, a native-runtime benchmark of 60 human-authored, bilingual, multimodal tasks spanning six thematic categories. Each task averages roughly 8 minutes of wall-clock time and over 20 tool calls, and runs inside a reproducible Docker container hosting an actual CLI agent harness (OpenClaw, Claude Code, Codex, or Hermes Agent) with access to real tools rather than mock services. Grading is hybrid, combining deterministic rule-based checks, environment-state auditing of side effects, and an LLM/VLM judge for semantic verification. Across 19 frontier models, the best, Claude Opus 4.7, reaches only 62.2% overall under OpenClaw, while every other model stays below 60%, and switching harness alone shifts a single model by up to 18 points. These results show that long-horizon, native-runtime agent evaluation remains a far-from-resolved task for current frontier models. We release the tasks, code, and containerized tooling to support reproducible evaluation.
Engineering Breakdown
Plain English
WildClawBench is a benchmark for evaluating AI agents on realistic, long-running tasks in actual production environments rather than synthetic sandboxes. It contains 60 real-world tasks averaging 8 minutes each with 20+ tool calls, running in Docker containers with genuine CLI tools instead of mocked APIs, testing agents like Claude Code and Codex on their ability to handle multi-step workflows that mirror actual user scenarios.
Key Engineering Insight
Most agent benchmarks measure final-answer accuracy on short tasks with fake APIs, but real deployed agents fail silently on long-horizon workflows due to tool invocation errors, context limits, and state management issues. This benchmark forces agents to survive actual runtime environments where tasks take minutes and require dozens of sequential decisions.
Why It Matters for Engineers
If you're shipping an AI agent product, synthetic benchmarks don't tell you if it will work when a user actually deploys it on their machine. WildClawBench reveals whether your agent can maintain state across 20+ tool calls, recover from errors, and complete tasks that take 8+ minutes—exactly the failure modes you'll encounter in production.
Research Context
Prior agent benchmarks (like ReAct, SWE-Bench) use constrained environments, mock APIs, and single-step evaluations, which don't expose integration failures or cumulative error propagation. WildClawBench advances the field by running agents in real Docker containers with actual tools, bridging the gap between what benchmarks measure and what production deployments actually demand.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::