Mobile GUI Agent Privacy Personalization with Trajectory Induced Preference Optimization
| Authors | Zhixin Lin et al. |
| Year | 2026 |
| HF Upvotes | 12 |
| arXiv | 2604.11259 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Mobile GUI agents powered by Multimodal Large Language Models (MLLMs) can execute complex tasks on mobile devices. Despite this progress, most existing systems still optimize task success or efficiency, neglecting users' privacy personalization. In this paper, we study the often-overlooked problem of agent personalization. We observe that personalization can induce systematic structural heterogeneity in execution trajectories. For example, privacy-first users often prefer protective actions, e.g., refusing permissions, logging out, and minimizing exposure, leading to logically different execution trajectories from utility-first users. Such variable-length and structurally different trajectories make standard preference optimization unstable and less informative. To address this issue, we propose Trajectory Induced Preference Optimization (TIPO), which uses preference-intensity weighting to emphasize key privacy-related steps and padding gating to suppress alignment noise. Results on our Privacy Preference Dataset show that TIPO improves persona alignment and distinction while preserving strong task executability, achieving 65.60% SR, 46.22 Compliance, and 66.67% PD, outperforming existing optimization methods across various GUI tasks. The code and dataset will be publicly released at https://github.com/Zhixin-L/TIPO.
Engineering Breakdown
Plain English
This paper addresses personalization in mobile GUI agents powered by multimodal large language models (MLLMs), focusing on how user preferences for privacy versus utility create fundamentally different execution trajectories on mobile devices. The authors observe that privacy-first users take protective actions like refusing permissions and logging out, while utility-first users optimize for task completion, resulting in variable-length and structurally heterogeneous execution paths. Standard preference optimization methods struggle with this heterogeneity because they assume relatively similar trajectories across users. The paper proposes a solution to handle this structural diversity, making agent personalization both stable and informative.
Core Technical Contribution
The core insight is recognizing that user personalization preferences induce systematic structural heterogeneity in execution trajectories—trajectories differ not just in length but in logical structure and action sequencing. Prior work treated personalization as a simple reward weighting problem, but this paper identifies that privacy-preference and utility-preference trajectories are fundamentally different execution patterns that cannot be optimized using standard preference learning methods. The technical novelty lies in developing an approach that explicitly handles variable-length, structurally heterogeneous trajectories without collapsing distinct user preferences into a single policy. This reframes agent personalization from a parameter-tuning problem into a structural trajectory modeling problem.
How It Works
The system takes as input a mobile GUI state (visual frame) and a user's personalization preference profile (privacy-first vs utility-first). An MLLM processes the visual input and preference context to generate action sequences that execute the task while respecting the user's preference. The key mechanism is handling trajectory heterogeneity: rather than training a single policy on mixed trajectories, the approach models different trajectory structures for different preference classes. For privacy-first users, the policy learns to include permission-denial actions, logout sequences, and data-minimization steps; for utility-first users, it learns direct task completion paths. The training likely uses demonstration data segregated by preference type, allowing the model to learn distinct decision policies. Output is a sequence of mobile UI actions (taps, text input, navigation) that complete the task while maintaining structural consistency with the user's preference class.
Production Impact
For teams building personalized mobile AI agents, this work shifts the paradigm from single-policy-fits-all to preference-aware policy learning, requiring you to collect or infer user preference labels from historical interaction data or explicit user settings. In production, this means storing user preference profiles alongside interaction logs, then either training separate model heads per preference class or using a conditional policy that branches on preference input. The trade-off is increased model complexity and training data requirements—you now need representative trajectories for each preference class rather than a single merged dataset. Latency impact is minimal (just preference conditioning), but you need infrastructure to manage multiple policy variants or conditional logic. For privacy-sensitive applications (banking, healthcare), this approach is high-value because it legitimizes and structures privacy preferences rather than treating them as optimization noise.
Limitations and When Not to Use This
The paper assumes user preferences are stable and well-defined, but in practice users may have context-dependent preferences or mixed objectives (privacy in some flows, utility in others) that don't fit clean categories. The approach requires sufficient trajectory data per preference class to learn distinct policies; rare preference classes or cold-start users with no historical data will suffer from poor personalization. The paper doesn't address how to infer or elicit user preferences reliably—explicit user input is burdensome, and inferring from behavior is error-prone and potentially manipulative. Additionally, the work focuses on mobile GUI agents and may not generalize to other domains with different action spaces, and it doesn't explore how structural heterogeneity scales with more than two preference dimensions (e.g., privacy + cost-consciousness + accessibility).
Research Context
This work builds on the rapidly growing field of MLLM-based agents (like Anthropic's Interleaver work, Google's Android agents) and extends preference learning techniques like RLHF and DPO beyond homogeneous trajectory distributions. It addresses a gap identified in recent mobile agent benchmarks (like MobileEnv, AITW) where generalization and user alignment are underexplored compared to task success metrics. The paper opens a new research direction: agent personalization as a structural problem rather than a scalar preference problem, potentially informing future work on multi-objective agent learning and user-agent alignment in safety-critical domains. This fits into the broader context of AI safety and alignment, where respecting diverse user values is critical for responsible deployment.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::