CocoaBench: Evaluating Unified Digital Agents in the Wild
| Authors | CocoaBench Team et al. |
| Year | 2026 |
| HF Upvotes | 33 |
| arXiv | 2604.11201 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
LLM agents now perform strongly in software engineering, deep research, GUI automation, and various other applications, while recent agent scaffolds and models are increasingly integrating these capabilities into unified systems. Yet, most evaluations still test these capabilities in isolation, which leaves a gap for more diverse use cases that require agents to combine different capabilities. We introduce CocoaBench, a benchmark for unified digital agents built from human-designed, long-horizon tasks that require flexible composition of vision, search, and coding. Tasks are specified only by an instruction and an automatic evaluation function over the final output, enabling reliable and scalable evaluation across diverse agent infrastructures. We also present CocoaAgent, a lightweight shared scaffold for controlled comparison across model backbones. Experiments show that current agents remain far from reliable on CocoaBench, with the best evaluated system achieving only 45.1% success rate. Our analysis further points to substantial room for improvement in reasoning and planning, tool use and execution, and visual grounding.
Engineering Breakdown
Plain English
CocoaBench introduces a unified benchmark for evaluating large language model agents that must combine multiple capabilities—vision, web search, and code execution—in a single workflow. The paper addresses a critical gap: most existing benchmarks test agent capabilities in isolation (e.g., only code generation or only visual reasoning), but real-world tasks require flexible composition of these skills. The authors created a benchmark with human-designed, long-horizon tasks specified by natural language instructions and automatic evaluation functions, enabling reproducible testing across different agent architectures. They also present CocoaAgent, a lightweight agent system designed to work effectively with this benchmark.
Core Technical Contribution
The core novelty is the benchmark design itself—moving from isolated capability evaluation to integrated multi-modal task composition. Rather than testing vision, search, and coding in separate silos, CocoaBench requires agents to seamlessly switch between these modes within a single task, forcing genuine integration of capabilities. The benchmark uses instruction-based task specification with automatic output evaluation, which provides scalable assessment without requiring human annotation at scale. This addresses a real gap in the agent evaluation landscape where existing benchmarks (SWE-bench for code, GAIA for reasoning, etc.) don't test the orchestration problem that defines production agent systems.
How It Works
CocoaBench tasks are structured as long-horizon workflows where an agent receives a natural language instruction and must navigate between multiple tools and modalities to complete it. The agent receives inputs through vision (screenshots, images), search (web APIs), and coding execution (Python sandboxes), and must decide when to invoke each capability. Each task has an automatic evaluation function that checks the final output (text, files, or structured data) against ground truth, eliminating manual annotation bottlenecks. The benchmark likely follows a pattern where an agent framework (like AutoGen or LangChain) orchestrates these tools, with the agent deciding which tool to invoke based on task state and progress. CocoaAgent appears to be a reference implementation—a lightweight scaffolding system that demonstrates how agents can effectively combine these modalities without requiring massive model sizes. The evaluation methodology enables rapid iteration by running many tasks in parallel and automatically grading results without human involvement.
Production Impact
For engineers building production agent systems, this benchmark provides a realistic testing ground that mirrors actual deployment scenarios where agents must juggle vision, search, and code. Currently, teams evaluate these capabilities separately, miss integration bugs, and deploy agents that fail in unexpected ways when real-world tasks require mode-switching. Adopting CocoaBench would catch composition failures early—scenarios like 'agent gets stuck in vision-only mode' or 'fails to search before coding.' The automatic evaluation function is critical for production: it enables continuous benchmarking as you iterate on your agent scaffold, similar to how test suites work for traditional software. Trade-offs include the overhead of setting up evaluation infrastructure (automatic grading functions need careful design), potential coverage gaps if tasks don't represent your specific use cases, and the need to maintain multiple execution backends (vision models, search APIs, code sandboxes) in your testing pipeline.
Limitations and When Not to Use This
CocoaBench is limited by its task design—human-created tasks may not exhaustively cover real-world failure modes, and there's likely a ceiling to how well synthetic tasks capture production complexity. The automatic evaluation function assumes the final output can be checked programmatically, which breaks down for open-ended tasks like 'write a helpful blog post' where quality requires human judgment. The paper doesn't address how well results transfer across different agent frameworks or model sizes; a task optimized for GPT-4 may not be solvable by smaller models, making benchmark results potentially non-comparable. Additionally, the benchmark likely doesn't address adversarial inputs, malicious tool responses, or the cost-quality tradeoffs that dominate production decisions (which search API to use, how many retries, fallback strategies). Follow-up work would need to handle partial credit (tasks with degrees of success), multi-turn interactions where agents must ask clarifying questions, and evaluation under resource constraints (latency budgets, token limits).
Research Context
This work builds on a recent shift in agent research from narrow benchmarks (SWE-bench for code, GAIA for reasoning, WebShop for shopping) toward unified evaluation frameworks that test real-world integration. It follows trends in papers like HELM (which evaluates language models across diverse tasks) and AutoGPT/ReAct work that showed agents need flexible tool composition. CocoaBench likely positions itself as the first benchmark specifically designed for testing this composition problem in agents—prior work either tested single modalities or tested them sequentially rather than requiring fluent switching. The paper opens a research direction around agent orchestration and evaluation methodology: how do we design benchmarks that scale with agent complexity, what makes a good automatic evaluation function, and how do we measure agent reliability across heterogeneous capabilities.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::