WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environments
| Authors | Jinchao Li et al. |
| Year | 2026 |
| HF Upvotes | 14 |
| arXiv | 2604.27776 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
While GUI agents have shown impressive capabilities in common computer-use tasks such as OSWorld, current benchmarks mainly focus on isolated and single-application tasks. This overlooks a critical real-world requirement of coordinating across multiple applications to accomplish complex profession-specific workflows. To bridge this gap, we present a computer-use benchmark in cross-application workflows, named WindowsWorld, designed to systematically assess GUI Agents on complex multi-step tasks that mirror real-world professional activities. Our methodology uses a multi-agent framework steered by 16 occupations to generate four difficulty-level tasks with intermediate inspection, which are then refined by human review and executed in a simulated environment. The resulting benchmark contains 181 tasks with an average of 5.0 sub-goals across 17 common desktop applications, of which 78% are inherently multi-application. Experimental results of leading large models and agents show that: 1) All computer-use agents perform poorly on multi-application tasks (< 21% success rate), far below the performance of simple single-app tasks; 2) They largely fail at tasks requiring conditional judgment and reasoning across geq 3 applications, stalling at early sub-goals; 3) Low execution efficiency, where tasks often fail despite far exceeding human step limits. Code, benchmark data, and evaluation resources are available at github.com/HITsz-TMG/WindowsWorld.
Engineering Breakdown
Plain English
This paper introduces WindowsWorld, a benchmark for evaluating GUI agents on complex cross-application workflows that mirror real-world professional tasks. Unlike existing benchmarks like OSWorld that focus on isolated single-application tasks, WindowsWorld presents multi-step scenarios spanning 16 different occupations with four difficulty levels. The benchmark was constructed using a multi-agent framework to generate diverse tasks, with human review and execution in a simulated Windows environment to ensure quality and realism. This addresses a critical gap in current AI evaluation: the ability to coordinate across multiple applications—a fundamental requirement for professional computer-use scenarios.
Core Technical Contribution
The core contribution is a systematic methodology for creating cross-application workflow benchmarks that captures real-world complexity beyond single-app isolation. Rather than manually hand-crafting tasks, the authors use a multi-agent framework steered by occupational personas to automatically generate diverse, difficulty-stratified scenarios with intermediate inspection points. This approach scales task creation across 16 professions and 4 difficulty levels while maintaining quality through human review and execution validation in a simulated environment. The novelty lies in recognizing that real professional work requires coordination between applications (e.g., copying data from a spreadsheet into an email, then scheduling a meeting), which existing benchmarks completely miss.
How It Works
The system operates through three main phases: generation, validation, and execution. First, a multi-agent framework uses 16 occupational personas (accountant, doctor, manager, etc.) to generate multi-step tasks that span multiple applications at four difficulty levels. The framework produces tasks with intermediate inspection points—checkpoints where the agent must verify state before proceeding—which ensures tasks are evaluable at granular steps. Second, human reviewers filter and refine the generated tasks to ensure they reflect realistic professional workflows and are solvable. Third, tasks execute in a simulated Windows environment where GUI agents attempt to complete them, with performance measured against the intermediate checkpoints and final outcomes. The benchmark measures not just whether agents reach the goal, but whether they correctly coordinate application-switching, data transfer, and state verification across the workflow.
Production Impact
For engineers building production GUI automation or AI assistant systems, this benchmark exposes a critical limitation in today's agent evaluation: single-application task performance does not predict cross-application competency. If you're deploying an agent to handle real customer workflows (like processing expense reports that require email, spreadsheet, and accounting software), you need this benchmark to validate that your system can actually perform the task. The practical impact is that you must now evaluate agents on multi-step, multi-application scenarios rather than relying on isolated task performance—this changes your evaluation pipeline and likely reveals gaps in your agent's state tracking and application-switching logic. The compute cost is primarily in the simulated environment execution; the human review cost is non-trivial but necessary to maintain benchmark quality. Latency-wise, cross-application workflows inherently take longer than single-app tasks, so you need realistic timing validation, not just binary success/failure.
Limitations and When Not to Use This
The paper's approach assumes that multi-agent framework generation plus human filtering produces representative real-world tasks, but this may miss edge cases that only emerge in production use across thousands of unique workflows. The benchmark is built for Windows environments and occupational personas—it may not generalize to macOS, Linux, or non-traditional workflows or industries not covered by the 16 personas. The simulated environment may not capture all the friction of real applications: network latency, authentication challenges, permission errors, or application crashes that occur in production. Additionally, the intermediate inspection points, while good for debugging, may not reflect how agents actually need to operate—in production, agents might need to recover from errors or adapt without clean checkpoints. The paper does not address how to handle dynamic environments where application state changes unexpectedly or where multiple valid solution paths exist.
Research Context
This work builds directly on GUI agent benchmarks like OSWorld (2024) but identifies and fixes a critical gap: OSWorld focuses on isolated, single-application tasks that don't reflect professional computer work. WindowsWorld extends the benchmarking tradition by raising the complexity to multi-application workflows, similar to how vision-language models moved from single-image classification to multi-image reasoning. The work also connects to broader research in embodied AI agents and task planning, where understanding how to coordinate across tools and applications is a fundamental challenge. This opens a new research direction: benchmarking agents not just on task completion but on their ability to maintain state, coordinate applications, and handle workflow dependencies—skills that will be essential as AI systems move from narrow task automation to general-purpose computer use.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::