OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesis
| Authors | Kanzhi Cheng et al. |
| Year | 2026 |
| HF Upvotes | 27 |
| arXiv | 2604.15093 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Mobile agents powered by vision-language models have demonstrated impressive capabilities in automating mobile tasks, with recent leading models achieving a marked performance leap, e.g., nearly 70% success on AndroidWorld. However, these systems keep their training data closed and remain opaque about their task and trajectory synthesis recipes. We present OpenMobile, an open-source framework that synthesizes high-quality task instructions and agent trajectories, with two key components: (1) The first is a scalable task synthesis pipeline that constructs a global environment memory from exploration, then leverages it to generate diverse and grounded instructions. and (2) a policy-switching strategy for trajectory rollout. By alternating between learner and expert models, it captures essential error-recovery data often missing in standard imitation learning. Agents trained on our data achieve competitive results across three dynamic mobile agent benchmarks: notably, our fine-tuned Qwen2.5-VL and Qwen3-VL reach 51.7% and 64.7% on AndroidWorld, far surpassing existing open-data approaches. Furthermore, we conduct transparent analyses on the overlap between our synthetic instructions and benchmark test sets, and verify that performance gains stem from broad functionality coverage rather than benchmark overfitting. We release data and code at https://njucckevin.github.io/openmobile/ to bridge the data gap and facilitate broader mobile agent research.
Engineering Breakdown
Plain English
OpenMobile is an open-source framework that trains mobile agents (AI systems that automate tasks on smartphones) by automatically generating high-quality training data. Current state-of-the-art models like those evaluated on AndroidWorld achieve ~70% success rates, but their training procedures and datasets remain proprietary. This paper introduces two key innovations: a scalable pipeline that builds environmental memory from exploration to generate diverse, grounded task instructions, and a policy-switching strategy during trajectory collection that alternates between learner and expert models to capture better decision-making patterns. The work addresses a critical gap in the field by providing transparency and reproducibility for mobile agent training.
Core Technical Contribution
The core novelty is a two-stage synthetic data generation approach for mobile agents that doesn't rely on human demonstrations or proprietary data. First, the task synthesis pipeline constructs a global environment memory by exploring mobile interfaces, then uses this memory to generate diverse and contextually grounded instructions—solving the problem of instruction diversity that limits generalization. Second, the policy-switching trajectory synthesis strategy alternates between a learner model and expert model during rollout, allowing the system to recover from early mistakes while still exploring diverse action sequences, which is more efficient than pure expert demonstrations or pure learning-from-scratch. This stands apart from prior work that either uses fixed datasets, relies on human annotation, or deploys single policies without adaptive switching.
How It Works
The system operates in two interconnected phases. Phase 1 (Task Synthesis): The framework explores mobile environments through a learner agent, building a global memory bank of screens, UI elements, and successful action sequences. It then seeds task generation with natural language prompts and leverages the environment memory to ground abstract instructions in the actual interfaces available—ensuring generated tasks are feasible and diverse rather than template-based repetitions. Phase 2 (Trajectory Synthesis): During rollout, the system implements a policy-switching mechanism where it follows a learner policy but can switch to an expert policy when the learner makes errors or reaches dead ends, then returns to learning mode. This creates trajectories that are both realistic (learner makes some mistakes) and high-quality (expert recovers), capturing the kind of recovery patterns needed for robust agents. The output is a paired dataset of (instruction, trajectory) tuples that can train the next generation of the model, creating a flywheel effect.
Production Impact
For teams building mobile automation systems, this framework eliminates dependency on proprietary training data and enables reproducible, scalable agent development. Instead of collecting thousands of human demonstrations or licensing closed datasets, engineers can now spin up environment exploration, automatically synthesize diverse tasks, and generate rollouts using open models—reducing time-to-production for domain-specific agents. The policy-switching strategy is particularly valuable in production because real users expect agents to recover gracefully from errors; this approach bakes recovery patterns into training. Trade-offs are significant: the approach requires computational resources to explore environments and run two models during trajectory generation (increasing training wall-clock time), and the quality of generated instructions depends heavily on the seed prompts and environment diversity. Integration complexity is moderate—teams need to define environment interfaces and reward signals for the expert policy, but the framework abstracts the exploration and synthesis logic.
Limitations and When Not to Use This
The paper does not address how performance scales when target environments are significantly different from training environments, limiting generalization claims. The policy-switching strategy requires a reliable expert policy, but the paper does not clarify how to obtain or bootstrap the expert when domain expertise is sparse or environment dynamics are complex. Real mobile environments contain ambiguous UI elements, overlapping clickable regions, and dynamic content that may not be captured adequately by environment memory built from exploration alone. Additionally, the approach assumes task instructions can be meaningfully grounded in environment memory, but fails gracefully when new UI patterns or app types appear post-training. The paper lacks empirical ablations on key design choices—such as the memory capacity, switching frequency, or instruction diversity metrics—making it unclear which components drive performance gains.
Research Context
This work builds on the recent wave of vision-language model (VLM) based mobile agents, extending systems like those benchmarked on AndroidWorld by addressing the reproducibility and data transparency crisis in the field. Prior work either relied on closed training recipes (leading to non-reproducible results) or used human demonstrations at scale (expensive and slow). OpenMobile advances the data synthesis direction pioneered by work on automatic curriculum generation and synthetic data for vision tasks, but applies it specifically to the sequential decision-making setting of mobile agents. This opens a new research direction: can we use environment exploration and policy switching to bootstrap increasingly capable agents without human involvement, potentially creating self-improving systems? The work also positions open-source mobile agent research as competitive with proprietary systems, likely to accelerate adoption and benchmarking in the community.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::