KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation
| Authors | Tongbo Chen et al. |
| Year | 2026 |
| HF Upvotes | 41 |
| arXiv | 2604.08455 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Personalized mobile agents that infer user preferences and calibrate proactive assistance hold great promise as everyday digital assistants, yet existing benchmarks fail to capture what this requires. Prior work evaluates preference recovery from static histories or intent prediction from fixed contexts. Neither tests whether an agent can elicit missing preferences through interaction, nor whether it can decide when to intervene, seek consent, or remain silent in a live GUI environment. We introduce KnowU-Bench, an online benchmark for personalized mobile agents built on a reproducible Android emulation environment, covering 42 general GUI tasks, 86 personalized tasks, and 64 proactive tasks. Unlike prior work that treats user preferences as static context, KnowU-Bench hides the user profile from the agent and exposes only behavioral logs, forcing genuine preference inference rather than context lookup. To support multi-turn preference elicitation, it instantiates an LLM-driven user simulator grounded in structured profiles, enabling realistic clarification dialogues and proactive consent handling. Beyond personalization, KnowU-Bench provides comprehensive evaluation of the complete proactive decision chain, including grounded GUI execution, consent negotiation, and post-rejection restraint, evaluated through a hybrid protocol combining rule-based verification with LLM-as-a-Judge scoring. Our experiments reveal a striking degradation: agents that excel at explicit task execution fall below 50% under vague instructions requiring user preference inference or intervention calibration, even for frontier models like Claude Sonnet 4.6. The core bottlenecks are not GUI navigation but preference acquisition and intervention calibration, exposing a fundamental gap between competent interface operation and trustworthy personal assistance.
Engineering Breakdown
Plain English
This paper introduces KnowU-Bench, a new online benchmark for evaluating personalized mobile agents that must infer user preferences and decide when to proactively help. Unlike existing benchmarks that treat user preferences as fixed context, KnowU-Bench tests whether agents can actively elicit missing preferences through interaction and determine when to intervene, ask permission, or stay silent in real GUI environments. The benchmark covers 192 total tasks across three categories: 42 general GUI tasks, 86 personalized tasks, and 64 proactive tasks, all running on a reproducible Android emulation environment. This directly addresses a gap in current evaluation methods, which only measure static preference recovery or intent prediction rather than dynamic interaction and timing decisions in live mobile interfaces.
Core Technical Contribution
The core contribution is a shift from evaluating user preference understanding as a static supervised learning problem to evaluating it as an interactive, sequential decision-making problem. KnowU-Bench introduces the concept of hidden user profiles that agents must progressively discover through their interactions, rather than being given as input during evaluation. The benchmark operationalizes three distinct agent capabilities: completing GUI tasks with general knowledge, personalizing behavior based on inferred preferences, and making calibrated decisions about when to take proactive action versus seeking consent versus remaining silent. This represents a fundamental methodological advance—moving from offline benchmarks with fixed inputs to reproducible online benchmarks that simulate realistic agent-user interaction loops on actual mobile interfaces.
How It Works
The benchmark runs on an Android emulation environment that provides a reproducible, controllable GUI testing ground. For each task, an agent receives a goal state (e.g., 'send an email'), initial GUI context, and optionally a user profile with preferences that define how the task should be completed (e.g., recipient preferences, formatting preferences). The agent must decide: (1) what GUI actions to take to accomplish the task, (2) when it lacks information about user preferences and needs to elicit them through interaction (e.g., asking clarifying questions), and (3) when to proactively suggest actions versus waiting for explicit user input. The benchmark evaluates three distinct capability levels across the 192 tasks: general task completion (can the agent navigate and use mobile UI elements), preference-aware personalization (does it adapt behavior based on learned preferences), and proactive assistance calibration (does it intervene helpfully without being intrusive). Success is measured by whether the agent produces the correct final outcome while respecting user preferences and making appropriate timing decisions about intervention.
Production Impact
For engineers building real personalized mobile assistants, this benchmark reveals whether current approaches actually work in interactive settings where preferences must be discovered dynamically. In production, you would integrate this evaluation framework into your CI/CD pipeline to test not just whether your agent can complete tasks, but whether it learns from user feedback and adjusts its proactivity appropriately—critical for user trust and adoption. The benchmark exposes a major gap: most deployed mobile agents rely on pre-computed user profiles or explicit settings, but KnowU-Bench forces agents to handle incomplete information and make decisions about when to ask for it. This directly impacts product decisions—if your agent fails the proactive tasks, you know it will either annoy users by intervening constantly or frustrate them by being passive. Implementation would require building state management to track inferred preferences across interactions, reward signals for calibrated help (not just task completion), and GUI automation infrastructure to run reproducible tests; the Android emulation environment they provide reduces setup friction significantly.
Limitations and When Not to Use This
The benchmark evaluates agents in a controlled emulation environment, not real-world Android devices with variable network conditions, background processes, or users with genuinely unexpected preferences—so transfer to production may be limited. The paper does not clearly define how user preference distributions are generated or how diverse they are; if preferences follow narrow patterns, agents may overfit to the benchmark without truly learning to handle novel user behaviors. The benchmark requires precise GUI automation and state tracking, which is computationally expensive and may not scale to continuous evaluation on large agent fleets; there are no reported numbers on emulation latency, throughput, or infrastructure costs. The paper truncates its abstract before fully describing evaluation metrics, making it unclear how success is formally measured, whether partial credit is given, or how to trade off task completion accuracy against proactivity calibration—crucial details for reproducibility and comparison with future work.
Research Context
This work builds on a line of research in mobile agent evaluation and user preference modeling, extending beyond static benchmarks like UI automation datasets and preference prediction challenges. Prior work typically separated two problems: (1) intent prediction from fixed user context (treated as a classification problem) and (2) GUI task completion (treated as navigation/action planning). KnowU-Bench unifies these and adds a third dimension: preference elicitation through interaction, which connects to active learning and dialogue research. The benchmark advances the field of personalized agent evaluation similarly to how RLHF benchmarks advanced LLM alignment—by introducing interactive sequences and calibrated decision-making rather than pure prediction accuracy. This opens research directions in preference inference from limited interaction data, uncertainty quantification in agent decision-making, and multi-turn user-agent collaboration on mobile platforms.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::