ClawBench: Can AI Agents Complete Everyday Online Tasks?
| Authors | Yuxuan Zhang et al. |
| Year | 2026 |
| HF Upvotes | 247 |
| arXiv | 2604.08523 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
AI agents may be able to automate your inbox, but can they automate other routine aspects of your life? Everyday online tasks offer a realistic yet unsolved testbed for evaluating the next generation of AI agents. To this end, we introduce ClawBench, an evaluation framework of 153 simple tasks that people need to accomplish regularly in their lives and work, spanning 144 live platforms across 15 categories, from completing purchases and booking appointments to submitting job applications. These tasks require demanding capabilities beyond existing benchmarks, such as obtaining relevant information from user-provided documents, navigating multi-step workflows across diverse platforms, and write-heavy operations like filling in many detailed forms correctly. Unlike existing benchmarks that evaluate agents in offline sandboxes with static pages, ClawBench operates on production websites, preserving the full complexity, dynamic nature, and challenges of real-world web interaction. A lightweight interception layer captures and blocks only the final submission request, ensuring safe evaluation without real-world side effects. Our evaluations of 7 frontier models show that both proprietary and open-source models can complete only a small portion of these tasks. For example, Claude Sonnet 4.6 achieves only 33.3%. Progress on ClawBench brings us closer to AI agents that can function as reliable general-purpose assistants.
Engineering Breakdown
Plain English
ClawBench introduces a new evaluation framework containing 153 realistic everyday online tasks spanning 144 live platforms across 15 categories—like making purchases, booking appointments, and submitting job applications. The benchmark tests AI agents on capabilities that go beyond existing benchmarks: extracting information from user-provided documents, navigating complex multi-step workflows across diverse platforms, and performing write-heavy operations requiring accurate form completion. This addresses a critical gap in AI agent evaluation, moving from simplified sandbox environments to real-world task complexity that reflects what users actually need automated. The paper establishes a concrete, measurable standard for assessing whether modern AI agents can handle the routine digital busywork that consumes significant time in everyday life and work.
Core Technical Contribution
The core contribution is ClawBench itself—a systematic evaluation framework that bridges the gap between existing agent benchmarks (which tend to be narrow or simulated) and production-grade real-world complexity. Rather than proposing a new algorithm or architecture, the authors engineered a comprehensive task suite grounded on 144 live, production websites, requiring agents to handle document understanding, cross-platform navigation, and form-filling at scale. This is novel because prior benchmarks (like MindAct or WebArena) either use sandboxed environments, synthetic tasks, or single-domain focus; ClawBench demands agents generalize across genuinely diverse platforms and handle the messiness of real HTML, varying UX patterns, and information extraction from user documents. The framework also establishes baseline performance metrics for state-of-the-art LLM-based agents, creating a reproducible standard against which future agent systems can be measured.
How It Works
ClawBench operates as a task collection and evaluation harness for autonomous AI agents. The input to an agent consists of a natural language instruction (e.g., 'book a flight to New York on March 15'), optional user-provided documents (PDFs, CSVs, screenshots), and access to live web platforms via a browser automation API. The agent must parse the instruction, locate and extract relevant information from provided documents, navigate through potentially multi-step workflows (e.g., login → search → filter → checkout), fill out forms with extracted or inferred data, and execute transactions or submissions. The evaluation metric is task success: whether the agent achieved the stated goal on the live platform without human intervention. The framework includes task categorization (purchase, booking, application, etc.) and tracks failure modes—such as information extraction errors, navigation errors, or form-filling mistakes—to diagnose where agents struggle. The benchmark uses existing agent architectures (likely Vision-Language Models with web navigation capabilities) without modification, making it a pure evaluation tool that reveals gaps in current agent abilities.
Production Impact
For engineers building AI agent systems, ClawBench provides a concrete, reproducible standard for evaluating whether an agent is production-ready for real-world automation. Instead of relying on marketing claims or narrow task suites, teams can measure success against 153 diverse, real-world scenarios and identify specific failure modes (e.g., document extraction on PDFs, form-field detection on unfamiliar sites). Adoption would require integrating a live-platform evaluation pipeline into your continuous testing infrastructure—which adds infrastructure complexity (managing bot detection, rate limits, potential account blocks on partner sites) and latency (real HTTP requests, not mocks). The benchmark reveals that current agents likely fail on 30–60% of tasks, indicating that moving an agent to production requires not just algorithm improvements but also module-level hardening on document parsing, element detection, and error recovery. Teams investing in agent systems should use ClawBench results to prioritize which capabilities (form-filling robustness, information extraction accuracy) to optimize first, rather than chasing undifferentiated model scaling.
Limitations and When Not to Use This
ClawBench measures success on current live platforms, which means the benchmark will degrade over time as websites redesign—tests that pass today may fail next quarter due to CSS changes or new authentication schemes, requiring continuous maintenance. The framework assumes agents have reliable access to live platforms without being blocked by CAPTCHA, rate-limiting, or anti-bot measures; in production, many sites aggressively prevent automation, making these tasks infeasible at scale without platform cooperation. The benchmark does not address multi-user or real-time task coordination (e.g., booking a meeting when calendar availability must be negotiated), nor does it test agent transparency, auditability, or the ability to explain decisions to users—critical for enterprise adoption. Finally, ClawBench measures task success as binary (completed or not) without measuring efficiency, cost, or robustness to minor variations in task phrasing, meaning high benchmark scores may not translate to reliable production automation when user instructions are noisy or incomplete.
Research Context
ClawBench extends a growing line of work on agent evaluation benchmarks—building on WebArena (simulated e-commerce), Mind2Web (web task diversity), and MindAct (agent reasoning)—but with a crucial shift toward live platforms and realistic task complexity. The paper responds to the observation that sandbox-based benchmarks often overestimate agent capability by allowing access to clean, predictable HTML; live platforms introduce real distribution shift, requiring agents to be robust to layout variation and data inconsistency. This work opens a new research direction: understanding why current agents fail on everyday tasks and what architectural or training improvements are needed—likely pointing toward better document-understanding modules, improved few-shot learning for form-filling, and more robust navigation strategies. ClawBench will likely become a standard benchmark in the agent literature, similar to how MNIST or SuperGLUE defined evaluation for vision and NLP; future papers will cite success rates on ClawBench as proof that an agent method works on realistic problems.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::