GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflows
| Authors | Jize Wang et al. |
| Year | 2026 |
| HF Upvotes | 1 |
| arXiv | 2604.15715 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
The development of general-purpose agents requires a shift from executing simple instructions to completing complex, real-world productivity workflows. However, current tool-use benchmarks remain misaligned with real-world requirements, relying on AI-generated queries, dummy tools, and limited system-level coordination. To address this, we propose GTA-2, a hierarchical benchmark for General Tool Agents (GTA) spanning atomic tool use and open-ended workflows. Built on real-world authenticity, it leverages real user queries, deployed tools, and multimodal contexts. (i) GTA-Atomic, inherited from our prior GTA benchmark, evaluates short-horizon, closed-ended tool-use precision. (ii) GTA-Workflow introduces long-horizon, open-ended tasks for realistic end-to-end completion. To evaluate open-ended deliverables, we propose a recursive checkpoint-based evaluation mechanism that decomposes objectives into verifiable sub-goals, enabling unified evaluation of both model capabilities and agent execution frameworks (i.e., execution harnesses). Experiments reveal a pronounced capability cliff: while frontier models already struggle on atomic tasks (below 50%), they largely fail on workflows, with top models achieving only 14.39% success. Further analysis shows that checkpoint-guided feedback improves performance, while advanced frameworks such as Manus and OpenClaw substantially enhance workflow completion, highlighting the importance of execution harness design beyond the underlying model capacity. These findings provide guidance for developing reliable personal and professional assistants. Dataset and code will be available at https://github.com/open-compass/GTA.
Engineering Breakdown
Plain English
This paper introduces GTA-2, a hierarchical benchmark for evaluating general-purpose AI agents on tool use tasks, addressing a critical gap in how agents are currently tested. The benchmark has two components: GTA-Atomic for short-horizon, closed-ended tool-use precision (inherited from prior work), and GTA-Workflow for long-horizon, open-ended realistic productivity workflows. Unlike existing benchmarks that rely on AI-generated queries, dummy tools, and limited system coordination, GTA-2 uses real user queries, deployed production tools, and multimodal contexts to reflect actual real-world requirements. This shift from instruction-following to authentic workflow completion represents a significant step toward evaluating agents that can handle the complexity of actual productivity software environments.
Core Technical Contribution
The core novelty is the introduction of GTA-Workflow, which extends tool-use evaluation beyond isolated atomic actions to realistic end-to-end multi-step workflows with genuine system-level coordination requirements. Rather than inventing a new algorithm, the authors identified a critical misalignment in how the field benchmarks agent capabilities—existing benchmarks optimize for simple tool-calling precision but ignore the composition, sequencing, and error recovery needed in real production systems. The key contribution is empirical and methodological: demonstrating that a hierarchical benchmark structure (atomic + workflow) using authentic data sources (real queries, production tools, multimodal I/O) better correlates with actual agent deployment success. This represents a shift from synthetic evaluation to authentic real-world validation, which is harder to game and more predictive of downstream performance.
How It Works
GTA-2 operates as a two-tier evaluation framework: the first tier (GTA-Atomic) maintains short-horizon, closed-ended tool-use tasks where agents must select the correct tool and parameters given a query, measuring precision in isolated action selection. The second tier (GTA-Workflow) presents long-horizon tasks that require agents to chain multiple tool calls across steps, handle dependencies between actions, manage intermediate state, and recover from failures—all while reasoning about the appropriate next action given the current application context. The benchmark ingests three input modalities: real user queries (extracted from production logs rather than synthetic templates), deployed tool APIs (actual production services with real constraints and behaviors), and multimodal contexts (screenshots, document state, application UI). The evaluation measures both task completion accuracy (did the agent achieve the goal?) and path efficiency (did it take a reasonable number of steps?), capturing both correctness and practical usability. This hierarchical structure allows separate measurement of atomic tool-calling ability versus workflow orchestration ability, which are distinct competencies in production agents.
Production Impact
For teams building agent systems, adopting GTA-2-style evaluation practices means moving from benchmark scores that don't predict real-world success toward metrics grounded in actual deployment scenarios. If you're building a productivity agent, you'd immediately benefit from testing against real tool APIs and authentic user queries rather than mocked ones, because this catches integration surprises, API brittleness, and state-management bugs that synthetic benchmarks miss. The workflow component is particularly valuable for multi-step tasks (e.g., 'draft an email, schedule a meeting, and update a spreadsheet')—common patterns in office automation—where agents must handle multi-action planning, error recovery, and context persistence. On the trade-off side, implementing or testing against GTA-2 requires maintaining real tool integrations and live API access, increasing infrastructure complexity and potentially introducing latency variability during evaluation. The multimodal context requirement (processing screenshots or UI state) also increases evaluation computational cost compared to text-only benchmarks, potentially 2-5x depending on vision encoder choice.
Limitations and When Not to Use This
The paper does not address how to handle adversarial or out-of-distribution inputs—real-world users attempt workflows that violate API preconditions or request unsafe actions, and GTA-2's evaluation likely focuses on happy paths. The benchmark's reliance on real deployed tools means evaluation cost and reproducibility are high; researchers without access to those specific SaaS platforms cannot reproduce results, limiting research velocity and democratization. The paper doesn't clearly define how workflow success is graded when there are multiple valid execution paths—if two agents complete a task with different sequences of tool calls, how is equivalence determined? This ambiguity could lead to inconsistent scoring. Finally, the abstract hints at multimodal evaluation but doesn't specify the vision model used, how screenshots are captured, or how frequently the UI context is updated during long-horizon workflows—key details needed to understand whether visual reasoning or simple state tracking dominates performance.
Research Context
This work builds on the original GTA benchmark and addresses a recognized limitation in the agent evaluation literature: existing benchmarks (like API-Bank, T-Bench, or ToolBench) typically use synthetic queries and isolated tool calls, failing to measure how well agents compose actions into real workflows. GTA-2 aligns with a broader trend toward more authentic evaluation (similar to BrowserAgent or WebArena in web automation), recognizing that benchmark realism directly correlates with production relevance. The hierarchical two-tier structure echoes evaluation approaches in robotics (atomic vs. task-level performance) and system testing (unit vs. integration testing), bringing those mature practices into the agent domain. This work opens research directions into workflow planning, multi-step error recovery, tool-result interpretation under incomplete/ambiguous feedback, and long-horizon credit assignment—problems largely absent from single-action tool-use datasets.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::