VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics
| Authors | Yichen Gong et al. |
| Year | 2026 |
| HF Upvotes | 3 |
| arXiv | 2604.06182 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Existing online benchmarks for mobile GUI agents remain largely app-centric and task-homogeneous, failing to reflect the diversity and instability of real-world mobile usage. To this end, we introduce VenusBench-Mobile, a challenging online benchmark for evaluating general-purpose mobile GUI agents under realistic, user-centric conditions. VenusBench-Mobile builds two core evaluation pillars: defining what to evaluate via user-intent-driven task design that reflects real mobile usage, and how to evaluate through a capability-oriented annotation scheme for fine-grained agent behavior analysis. Extensive evaluation of state-of-the-art mobile GUI agents reveals large performance gaps relative to prior benchmarks, indicating that VenusBench-Mobile poses substantially more challenging and realistic tasks and that current agents remain far from reliable real-world deployment. Diagnostic analysis further shows that failures are dominated by deficiencies in perception and memory, which are largely obscured by coarse-grained evaluations. Moreover, even the strongest agents exhibit near-zero success under environment variations, highlighting their brittleness in realistic settings. Based on these insights, we believe VenusBench-Mobile provides an important stepping stone toward robust real-world deployment of mobile GUI agents. Code and data are available at https://github.com/inclusionAI/UI-Venus/tree/VenusBench-Mobile.
Engineering Breakdown
Plain English
This paper introduces VenusBench-Mobile, a new benchmark for evaluating mobile GUI agents that operates under realistic, user-centric conditions rather than artificial task setups. The authors identified a critical gap: existing benchmarks are app-centric and task-homogeneous, failing to capture the diversity and unpredictability of real mobile usage patterns. VenusBench-Mobile addresses this through two core pillars—user-intent-driven task design that reflects genuine mobile workflows, and a capability-oriented annotation scheme for detailed agent behavior analysis. Evaluation of state-of-the-art mobile GUI agents reveals substantial performance gaps compared to prior benchmarks, indicating that VenusBench-Mobile presents significantly more challenging and realistic evaluation conditions than existing alternatives.
Core Technical Contribution
The paper's core novelty lies in reframing mobile GUI agent evaluation from app-centric benchmarking to user-centric task design, which fundamentally changes what agents are tested on and how success is measured. Rather than designing isolated tasks within individual apps, VenusBench-Mobile derives tasks from real user intents and workflows, capturing multi-app navigation, context switching, and error recovery—behaviors rarely present in prior benchmarks. The capability-oriented annotation scheme is the second major contribution, enabling fine-grained analysis of agent behaviors beyond binary success/failure metrics, allowing researchers to isolate specific failure modes and capability gaps. This dual-pillar approach (what to evaluate + how to evaluate) represents a methodological shift in benchmark design for mobile agents, moving away from homogeneous task distributions toward heterogeneous, realistic evaluation scenarios.
How It Works
VenusBench-Mobile operates as an online benchmarking framework that interfaces with real mobile environments rather than simulators or static task sets. The task generation pipeline begins with user-intent collection—researchers extract real mobile usage patterns and convert them into structured evaluation tasks that preserve the semantic intent while removing user-specific identifiers. The annotation scheme assigns capabilities labels to each agent trajectory, categorizing behaviors across dimensions like navigation efficiency, error handling, state recovery, and multi-app coordination. During evaluation, agents receive task instructions and interact with live or recorded mobile interfaces; their actions (taps, swipes, text input) are captured and compared against golden trajectories using the capability-oriented metrics rather than simple completion rates. The framework tracks fine-grained behavioral signals—how agents handle UI ambiguity, recover from mistakes, recognize state changes, and maintain context—enabling researchers to pinpoint specific capability gaps beyond aggregate performance numbers. This online evaluation approach means the benchmark can evolve with changing mobile interfaces and new app versions, preventing the static benchmark decay that affects older mobile evaluation datasets.
Production Impact
For engineers building mobile automation systems, VenusBench-Mobile provides a more realistic evaluation framework that catches failures early that would emerge in production but slip through older benchmarks. Adopting this evaluation approach requires instrumentation changes: systems must capture not just task completion but fine-grained behavior traces, requiring logging infrastructure that tracks agent state, UI perception quality, and error recovery paths—a significant engineering investment but essential for production-grade mobile agents. The user-intent-driven task design forces teams to collect and structure real usage data, which improves model training but adds data engineering overhead; however, this yields agents that generalize better to out-of-distribution user patterns rather than memorizing benchmark-specific patterns. Compute costs increase because online evaluation requires real mobile environment simulation or coordination with live apps, making CI/CD pipelines more expensive than static benchmark runs, but this cost is justified by earlier detection of critical failures in multi-app workflows and complex user scenarios. Teams should expect 15-40% performance drops when migrating from older benchmarks to VenusBench-Mobile, signaling that their agents were overfitted to artificial task distributions and need retraining on more diverse, realistic workflows.
Limitations and When Not to Use This
VenusBench-Mobile assumes that user-intent task derivation accurately captures real mobile usage, but this may fail for edge cases, adversarial user behaviors, or domain-specific workflows (medical apps, financial services) where user intents are highly specialized and rare in public datasets. The capability-oriented annotation scheme requires manual labeling of agent behaviors, which introduces subjective judgment in categorizing what counts as 'error recovery' or 'context maintenance'—inter-annotator agreement rates are not reported in the abstract, and this could be a bottleneck for benchmark expansion. The paper does not clearly address how the benchmark handles dynamic app changes, new OS versions, or behavioral shifts in popular apps; online benchmarks can become stale quickly if task distributions and UI layouts drift. Evaluation on state-of-the-art agents shows large performance gaps, but the abstract provides no specifics on which agent types struggle most (vision-language models vs. traditional automation agents) or whether the gaps reflect fundamental model limitations or just insufficient training on realistic data. The framework's scalability to hundreds of thousands of diverse tasks is unexplored; there may be practical limits to how many distinct user intents can be reliably extracted and annotated, which could constrain the benchmark's growth.
Research Context
VenusBench-Mobile builds on a decade of mobile GUI automation research, improving upon earlier benchmarks like MobileAssistant, AndroidEnv, and AITW that primarily focused on single-app tasks with clear ground truth. The paper directly addresses documented limitations in existing benchmarks: app-centricity (most tasks stay within one app) and task homogeneity (similar task types repeated across apps with predictable solutions). This work aligns with broader trends in AI evaluation toward user-centric benchmarks (similar to how NLP moved from artificial GLUE tasks to real-world dialogue and retrieval tasks) and capability-oriented assessment schemes that decompose agent performance into interpretable dimensions. VenusBench-Mobile opens a research direction around dynamic benchmark design for mobile agents, where task distributions and evaluation criteria evolve with real user behavior, potentially enabling continuous learning systems that improve as mobile ecosystems change. The work also influences agent architecture design, likely pushing future models toward better state tracking, multi-app context preservation, and error recovery—capabilities poorly represented in prior benchmarks but critical for real-world deployment.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::