PAGER: Bridging the Semantic-Execution Gap in Point-Precise Geometric GUI Control
:::info Stub — Full Engineering Breakdown Coming This paper has a linked code implementation and was featured on Hugging Face Papers with 6 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Jingxuan Wei et al. |
| Year | 2026 |
| HF Upvotes | 6 |
| arXiv | 2605.15963 |
| Download | |
| Code | https://github.com/OpenRaiser/Pager |
Abstract
Large vision-language models have significantly advanced GUI agents, enabling executable interaction across web, mobile, and desktop interfaces. Yet these gains largely rely on a forgiving region-tolerant paradigm, where many nearby pixels inside the same component remain valid. Precise geometric construction breaks this assumption: actions must land on points in continuous canvas space rather than tolerant regions. Because geometric primitives carry ontological dependencies, a local coordinate error can induce cascading topological failures that distort downstream objects and invalidate the final construction. We identify this regime as precision-sensitive GUI tasks, requiring point-level accuracy, geometry-aware verification, and robustness to dependency-driven error propagation. To benchmark it, we introduce PAGE Bench, with 4,906 problems and over 224K process-supervised, pixel-level GUI actions. We further propose PAGER, a topology-aware agent that decomposes construction into dependency-structured planning and pixel-level execution. Pixel-grounded supervised tuning establishes executable action grammar, while precision-aligned reinforcement learning mitigates rollout-induced exposure bias through state-conditioned geometric feedback. Experiments reveal a pronounced Semantic-Execution Gap: general multimodal models can exceed 88% action type accuracy yet remain below 6% task success. PAGER closes this gap, delivering 4.1x higher task success than the strongest evaluated general baseline and raising step success rate from below 9% for GUI-specialized agents to over 62%, establishing a new state of the art for point-precise GUI control.
Engineering Breakdown
Plain English
This paper identifies a critical failure mode in vision-language model-based GUI agents: while current models work well when clicks can land anywhere within a UI component (forgiving), they fail catastrophically on precision-sensitive tasks like geometric drawing where clicks must hit exact pixel coordinates. The researchers show that small coordinate errors in geometric primitives cascade through dependent objects, breaking entire constructions, and propose PAGER to handle point-level accuracy and error propagation robustness.
Key Engineering Insight
The core insight is that GUI automation has two fundamentally different regimes — region-tolerant (web/mobile clicks) and precision-sensitive (CAD, drawing tools) — and existing VLM agents trained on the former fail completely on the latter because local errors propagate through geometric dependencies rather than staying isolated.
Why It Matters for Engineers
If you're building GUI automation systems, this matters because you likely assumed clicks on buttons or inputs just need to be 'close enough.' But the moment you need to support design tools, spreadsheets with precise cell selection, or any canvas-based interaction, your vision model's coordinate predictions become a bottleneck that breaks in unpredictable ways due to error cascading.
Research Context
Prior GUI automation research focused on high-level task completion where pixel-perfect accuracy wasn't required. This paper extends the frontier to precision-sensitive domains where the topology of constructed objects depends on exact coordinates — a gap between what VLMs can do well (semantic understanding) and what they struggle with (geometric execution), enabling the next generation of AI assistants for professional tools.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::