UI-Copilot: Advancing Long-Horizon GUI Automation via Tool-Integrated Policy Optimization
| Authors | Zhengxi Lu et al. |
| Year | 2026 |
| HF Upvotes | 6 |
| arXiv | 2604.13822 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
MLLM-based GUI agents have demonstrated strong capabilities in complex user interface interaction tasks. However, long-horizon scenarios remain challenging, as these agents are burdened with tasks beyond their intrinsic capabilities, suffering from memory degradation, progress confusion, and math hallucination. To address these challenges, we present UI-Copilot, a collaborative framework where the GUI agent focuses on task execution while a lightweight copilot provides on-demand assistance for memory retrieval and numerical computation. We introduce memory decoupling to separate persistent observations from transient execution context, and train the policy agent to selectively invoke the copilot as Retriever or Calculator based on task demands. To enable effective tool invocation learning, we propose Tool-Integrated Policy Optimization (TIPO), which separately optimizes tool selection through single-turn prediction and task execution through on-policy multi-turn rollouts. Experimental results show that UI-Copilot-7B achieves state-of-the-art performance on challenging MemGUI-Bench, outperforming strong 7B-scale GUI agents such as GUI-Owl-7B and UI-TARS-1.5-7B. Moreover, UI-Copilot-7B delivers a 17.1% absolute improvement on AndroidWorld over the base Qwen model, highlighting UI-Copilot's strong generalization to real-world GUI tasks.
Engineering Breakdown
Plain English
This paper addresses a fundamental problem in multimodal large language model (MLLM) based GUI agents: they struggle with long-horizon tasks because they can't reliably remember past observations, get confused about progress, and make mathematical errors. The authors propose UI-Copilot, a framework that separates task execution from memory and computation concerns by having a main GUI agent work alongside a lightweight copilot that handles retrieval and math on-demand. The key innovation is memory decoupling, which keeps persistent task observations separate from temporary execution context, and a learned policy that decides when to invoke the copilot as either a retriever or calculator. This collaborative approach enables agents to handle complex, multi-step UI interactions that previously exceeded their capabilities.
Core Technical Contribution
The core technical novelty is memory decoupling combined with selective tool invocation learning—rather than forcing a single agent to handle memory, execution, and computation simultaneously, the authors decouple persistent observations from transient context and train the policy to recognize when it needs external help. This is fundamentally different from prior MLLM agents that operate monolithically; instead of expecting one model to do everything, UI-Copilot creates a two-stage architecture where the main agent focuses on understanding UI state and task flow while delegating memory retrieval and numerical reasoning to specialized, lightweight modules. The mechanism for learning when to invoke which tool is non-trivial: the policy must be trained to identify task demands that require external computation or retrieval, rather than hallucinating answers. This represents a shift from end-to-end monolithic reasoning to modular, collaborative reasoning where agents learn to know their limitations.
How It Works
The system operates with a primary GUI agent that processes visual observations from the UI and determines the next action, but instead of maintaining all context internally, it separates observations into two classes: persistent observations (long-term task state) and transient context (immediate execution state). When the agent encounters a task demand—identified by learned signals—it can invoke the copilot in one of two modes: as a Retriever that fetches relevant prior observations from memory when context becomes stale, or as a Calculator that performs exact numerical computation when the agent would otherwise hallucinate numbers. The policy learning mechanism trains the agent to recognize when it should invoke each mode, likely through a learned gate or decision function that examines the current task context and history. The copilot itself is lightweight, suggesting it may be a simpler model or even a programmatic module that doesn't require the full computational overhead of the main MLLM. At inference time, the agent makes decisions on-demand—querying memory only when needed and deferring computation rather than attempting all reasoning in-context—which reduces the memory burden on the main model.
Production Impact
For engineers building GUI automation systems, this approach directly solves three critical failure modes that plague current MLLM agents: memory degradation (forgetting early task context), progress confusion (losing track of what has been done), and hallucinated math (inventing numbers instead of computing them). Adopting this framework would change your architecture from a single monolithic agent to a coordinator-plus-specialists pattern, where you'd implement a memory store (likely a retrieval database), a lightweight computation module, and a decision layer that routes between them. The production benefits are significant: you can use smaller or cheaper models for the main agent since it no longer needs to maintain all state, and you get more interpretable failures (you see exactly when the agent asked for help) compared to silent hallucinations. Trade-offs include increased latency from tool invocation decisions and additional infrastructure for the copilot and memory store, plus the need to train the policy for selective invocation, which requires labeled examples of when tools should be used. For long-horizon tasks (10+ steps), this approach likely outperforms monolithic agents, but for simple one-step UI tasks, the overhead may not justify the complexity.
Limitations and When Not to Use This
The paper's approach assumes that memory degradation and computation errors are the primary failure modes in long-horizon tasks, but doesn't address other challenges like adversarial or unexpected UI changes, which may still confuse the agent regardless of memory quality. The effectiveness depends heavily on the quality of the retriever and calculator modules—if these components are weak or slow, the collaborative approach provides little benefit and just adds latency. The paper doesn't clarify how the policy is trained to recognize when to invoke the copilot: if this requires dense labels on training data indicating exactly where tools should be called, data collection becomes expensive and the approach doesn't generalize to novel task types. Additionally, there's no discussion of how the method scales to extremely long-horizon tasks (100+ steps) or multi-branching task trees where the decision space for tool invocation becomes combinatorially large. The lightweight copilot assumption may not hold in practice—if reliable retrieval or computation requires sophisticated reasoning, the copilot becomes another full MLLM, negating the efficiency gains.
Research Context
This work builds on the growing body of research in MLLM-based agents for UI understanding and automation, which has accelerated with models like GPT-4V and Gemini demonstrating strong visual reasoning. The paper contributes to the emerging pattern of decomposing monolithic LLM-based agents into modular systems with specialized tools—similar to the shift from end-to-end LLMs to LLMs-with-tools (like ReAct, Toolformer, and recent work on tool-using agents). The memory decoupling concept relates to broader research in retrieval-augmented generation (RAG) and contextualized memory for long-context reasoning, extending these ideas to the interactive agent domain. By introducing selective tool invocation as a learned behavior rather than hand-engineered logic, the work opens a research direction in learnable agent architectures that intelligently allocate reasoning across components, which could influence future work on efficient multi-agent and hierarchical reasoning systems.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::