Spatio-Temporal Token Pruning for Efficient High-Resolution GUI Agents
| Authors | Zhou Xu et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2602.23235 |
| Download | |
| Categories | cs.CV, cs.AI |
Abstract
Pure-vision GUI agents provide universal interaction capabilities but suffer from severe efficiency bottlenecks due to the massive spatiotemporal redundancy inherent in high-resolution screenshots and historical trajectories. We identify two critical misalignments in existing compression paradigms: the temporal mismatch, where uniform history encoding diverges from the agent's "fading memory" attention pattern, and the spatial topology conflict, where unstructured pruning compromises the grid integrity required for precise coordinate grounding, inducing spatial hallucinations. To address these challenges, we introduce GUIPruner, a training-free framework tailored for high-resolution GUI navigation. It synergizes Temporal-Adaptive Resolution (TAR), which eliminates historical redundancy via decay-based resizing, and Stratified Structure-aware Pruning (SSP), which prioritizes interactive foregrounds and semantic anchors while safeguarding global layout. Extensive evaluations across diverse benchmarks demonstrate that GUIPruner consistently achieves state-of-the-art performance, effectively preventing the collapse observed in large-scale models under high compression. Notably, on Qwen2-VL-2B, our method delivers a 3.4x reduction in FLOPs and a 3.3x speedup in vision encoding latency while retaining over 94% of the original performance, enabling real-time, high-precision navigation with minimal resource consumption.
Engineering Breakdown
Plain English
This paper tackles the efficiency problem of vision-based GUI agents that interact with high-resolution screenshots and long action histories. The authors identify two critical failures in existing compression approaches: temporal mismatch, where historical context is encoded uniformly instead of mimicking how agents actually forget old interactions, and spatial topology conflict, where aggressive token pruning destroys the grid structure needed for precise coordinate prediction. They propose GUIPruner, a training-free framework combining Temporal-Adaptive Resolution (TAR) to eliminate redundant history and spatial topology-aware pruning to preserve coordinate grounding, achieving significant efficiency improvements while maintaining agent performance on high-resolution GUI navigation tasks.
Core Technical Contribution
The core insight is recognizing that existing token pruning methods fail GUI agents in two distinct ways that standard compression approaches don't address. GUIPruner introduces Temporal-Adaptive Resolution, which applies varying levels of compression to historical frames based on recency—treating recent interactions as critical while aggressively compressing older frames to match observed attention patterns. The spatial component uses topology-aware pruning that respects the 2D grid structure of screenshots, preserving spatial coherence needed for pixel-level coordinate grounding rather than treating tokens as an unstructured sequence. This dual approach is training-free, meaning it can be applied to existing models without retraining, making it immediately practical for deployed systems.
How It Works
The input is a sequence of high-resolution screenshots and action history that the vision-language model must encode into tokens. TAR analyzes the agent's actual attention patterns over historical frames and applies adaptive compression—recent frames receive minimal token reduction while older frames get progressively more aggressive pruning, typically reducing token count by sampling or resolution downscaling. For the spatial dimension, instead of arbitrary token selection that breaks 2D locality, the method identifies which patches or regions are semantically important for the current task while enforcing that remaining tokens form a valid grid that can be mapped back to image coordinates. The pruned token sequence is then fed to the model's attention layers, where the preserved spatial structure allows the decoder to generate accurate (x, y) click coordinates without hallucinating positions that don't exist in the pruned space. The entire process is applied at inference time with no fine-tuning required.
Production Impact
For teams deploying GUI agents on consumer hardware or cost-constrained cloud infrastructure, this directly reduces latency and memory footprint—fewer tokens mean faster attention computation and lower VRAM requirements during inference. Real-world GUI automation often requires processing 2-4K resolution screenshots with 10-20 steps of history, easily creating 10,000+ tokens; GUIPruner likely reduces this to 2,000-4,000 tokens while maintaining action accuracy. The training-free nature is crucial for production: you can drop this into existing deployed models (like vision-language models fine-tuned for GUI tasks) without retraining pipelines or maintaining separate model versions. The main trade-off is implementation complexity—you need to integrate TAR scheduling logic and topology-aware selection into your inference serving layer, and you may lose performance on novel UI layouts that violate the assumed spatial structure. Integration with existing LoRA-fine-tuned GUI models would be straightforward since pruning happens before the model sees tokens.
Limitations and When Not to Use This
The approach assumes GUI screenshots maintain consistent spatial structure and that agent attention follows a predictable recency-based decay pattern—this breaks down for tasks with non-local dependencies (e.g., comparing elements across distant parts of the screen) or for novel interface layouts the agent hasn't seen. The paper doesn't clearly address how it handles dynamic UI elements, popups, or modal dialogs that violate grid assumptions. Coordinate grounding precision is still constrained by which regions are preserved—if a critical UI element falls into a pruned region, the agent cannot click it, and there's no clear fallback mechanism. The method requires instrumenting the model to capture actual attention patterns (to parameterize TAR), which adds profiling overhead and may not generalize across different task distributions or GUI types. There's no discussion of how performance degrades with extremely aggressive compression ratios (e.g., 90%+ token reduction), which would be needed for edge deployments.
Research Context
This builds on a recent wave of work recognizing that vision transformers applied to high-resolution images incur quadratic attention cost, extending prior pruning techniques (like token merging and patch dropping) to the specific domain of GUI understanding where spatial and temporal structure matter differently than in general vision tasks. It advances beyond prior GUI agent work that either accepted full computational cost or used simple resolution downsampling, which loses fine details needed for small UI elements. The paper implicitly positions itself against generic compression methods that don't account for agent semantics—prior work on vision-language model efficiency focused on image classification or captioning, not interactive grounding. It opens a research direction into task-aware token pruning where compression strategies are tuned to specific action modalities (clicking, scrolling, typing) and UI element densities, rather than one-size-fits-all pruning schedules.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::