On the Reliability of Computer Use Agents
| Authors | Gonzalo Gonzalez-Pumariega et al. |
| Year | 2026 |
| HF Upvotes | 11 |
| arXiv | 2604.17849 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Computer-use agents have rapidly improved on real-world tasks such as web navigation, desktop automation, and software interaction, in some cases surpassing human performance. Yet even when the task and model are unchanged, an agent that succeeds once may fail on a repeated execution of the same task. This raises a fundamental question: if an agent can succeed at a task once, what prevents it from doing so reliably? In this work, we study the sources of unreliability in computer-use agents through three factors: stochasticity during execution, ambiguity in task specification, and variability in agent behavior. We analyze these factors on OSWorld using repeated executions of the same task together with paired statistical tests that capture task-level changes across settings. Our analysis shows that reliability depends on both how tasks are specified and how agent behavior varies across executions. These findings suggest the need to evaluate agents under repeated execution, to allow agents to resolve task ambiguity through interaction, and to favor strategies that remain stable across runs.
Engineering Breakdown
Plain English
This paper investigates why computer-use agents that can successfully complete a task once often fail on repeated executions of the same task, even when nothing changes. The authors identify three root causes of this unreliability: stochasticity during execution (randomness in the agent's decisions), ambiguity in how tasks are specified, and inherent variability in agent behavior patterns. They conduct a systematic analysis on OSWorld, a benchmark for operating system tasks, using repeated task executions and statistical tests to isolate which factor contributes most to failures. The key finding is that even high-performing agents lack reliability—a critical gap between one-shot success and consistent performance that has been largely overlooked in the literature.
Core Technical Contribution
The paper's core contribution is a rigorous empirical framework for decomposing unreliability in computer-use agents into three independent, measurable factors. Rather than treating reliability as a monolithic problem, the authors introduce paired statistical testing methodology that isolates how stochasticity, task ambiguity, and behavioral variance each affect task success rates across repeated executions. This is novel because prior work focused on improving peak performance on held-out test sets, not on understanding why agents fail on deterministic reruns of tasks they previously solved. The framework provides the first systematic evidence that task-level variability and specification ambiguity are often more significant than execution randomness—a counterintuitive finding that challenges how we benchmark and develop these systems.
How It Works
The methodology works by executing the same computer-use task multiple times under controlled conditions and measuring failure modes. For each task, the agent receives identical inputs and initial states, but the authors systematically vary factors: they run the agent multiple times to measure execution stochasticity (does the same input produce different action sequences?), they modify task instructions to measure ambiguity sensitivity (do slightly different phrasings change success rates?), and they analyze decision traces to capture behavioral variance (does the agent make different high-level choices across runs even with the same specification?). The core technical approach uses OSWorld tasks—real operating system interactions like file management and software configuration—where ground truth success is objectively verifiable. They apply paired statistical tests (likely McNemar's test or similar) to measure whether differences in success rates across repeated executions are significant. The key insight is that they hold the agent and model constant and only vary execution context, isolating which source of variance explains failures.
Production Impact
For teams building production computer-use agents, this work directly impacts how you should design reliability safeguards and testing protocols. Instead of accepting agent failures as inevitable, you now have a framework to diagnose root causes: you can run your agent multiple times on the same task and use statistical testing to determine whether failures stem from model uncertainty, poor prompt engineering, or genuine task ambiguity. This changes your deployment strategy—if ambiguity is the culprit, you invest in better task specifications or interactive clarification; if execution stochasticity dominates, you need ensemble methods or deterministic action selection; if behavioral variance is the issue, you may need finetuning or constrained decoding. The latency cost is manageable (rerunning tasks offline for diagnosis), but the operational insight is substantial. Production systems should implement this diagnostic framework as part of continuous monitoring, flagging tasks with low reliability even after initial validation passes.
Limitations and When Not to Use This
This paper identifies the reliability problem but does not propose solutions to fix it—it is fundamentally diagnostic work. The analysis is limited to OSWorld tasks, which are synthetic and may not reflect the full complexity of real-world enterprise software or APIs with non-deterministic backends; results may not generalize to web agents interacting with dynamic content or time-dependent systems. The paper assumes that repeated task execution under identical conditions is feasible, which breaks down for tasks with side effects or irreversible state changes (e.g., deleting a file successfully means you cannot retry the same task). Additionally, the statistical testing methodology requires sufficient samples to detect significance, so for rare but critical failure modes, you may lack statistical power. The work does not address how to trade off task specification clarity against task complexity—overly detailed specs may eliminate ambiguity but become brittle to real-world variation.
Research Context
This paper builds on the rapid progress in vision-language models and agentic systems (Claude's computer use, GPT-4V integration with act/observe loops) but shifts focus from benchmark peak performance to operational reliability. It extends prior work on computer vision grounding and multimodal action generation by treating the agent-environment loop as a stochastic system requiring reliability analysis rather than just accuracy optimization. The work relates to emerging safety and robustness research in AI agents, acknowledging that deployment-ready systems must be trustworthy, not just capable. OSWorld serves as the primary evaluation benchmark, and this paper likely influences future benchmark design to include reliability metrics alongside task success rates, potentially shifting the field toward more thorough statistical characterization of agent behavior.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::