Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models
| Authors | Yuheng Shi et al. |
| Year | 2026 |
| HF Upvotes | 5 |
| arXiv | 2604.06912 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
MLLMs require high-resolution visual inputs for fine-grained tasks like document understanding and dense scene perception. However, current global resolution scaling paradigms indiscriminately flood the quadratic self-attention mechanism with visually redundant tokens, severely bottlenecking inference throughput while ignoring spatial sparsity and query intent. To overcome this, we propose Q-Zoom, a query-aware adaptive high-resolution perception framework that operates in an efficient coarse-to-fine manner. First, a lightweight Dynamic Gating Network safely bypasses high-resolution processing when coarse global features suffice. Second, for queries demanding fine-grained perception, a Self-Distilled Region Proposal Network (SD-RPN) precisely localizes the task-relevant Region-of-Interest (RoI) directly from intermediate feature spaces. To optimize these modules efficiently, the gating network uses a consistency-aware generation strategy to derive deterministic routing labels, while the SD-RPN employs a fully self-supervised distillation paradigm. A continuous spatio-temporal alignment scheme and targeted fine-tuning then seamlessly fuse the dense local RoI with the coarse global layout. Extensive experiments demonstrate that Q-Zoom establishes a dominant Pareto frontier. Using Qwen2.5-VL-7B as a primary testbed, Q-Zoom accelerates inference by 2.52 times on Document & OCR benchmarks and 4.39 times in High-Resolution scenarios while matching the baseline's peak accuracy. Furthermore, when configured for maximum perceptual fidelity, Q-Zoom surpasses the baseline's peak performance by 1.1% and 8.1% on these respective benchmarks. These robust improvements transfer seamlessly to Qwen3-VL, LLaVA, and emerging RL-based thinking-with-image models. Project page is available at https://yuhengsss.github.io/Q-Zoom/.
Engineering Breakdown
Plain English
This paper addresses a critical bottleneck in multimodal large language models (MLLMs) that need high-resolution image inputs for tasks like document understanding and scene perception. Current approaches naively scale resolution globally, which floods the self-attention mechanism with redundant visual tokens and kills inference speed. The authors propose Q-Zoom, a query-aware adaptive framework that intelligently decides when to process high-resolution details and when coarse features suffice, using a Dynamic Gating Network to skip expensive processing and a Self-Distilled Region Proposal Network to focus computation on task-relevant regions. This coarse-to-fine approach maintains accuracy while dramatically improving throughput by exploiting spatial sparsity and aligning processing with actual query intent.
Core Technical Contribution
The core innovation is decoupling resolution scaling from global image processing through query-aware adaptive selection. Instead of indiscriminately upscaling all images, Q-Zoom introduces two key mechanisms: a lightweight Dynamic Gating Network that acts as a router to bypass high-resolution processing when unnecessary, and a Self-Distilled Region Proposal Network that learns to propose only task-relevant regions without requiring external annotations. This differs fundamentally from prior work that treats resolution as a global parameter—Q-Zoom makes per-query, spatially-aware decisions about where and when to allocate computational budget. The self-distillation component is particularly novel, allowing the RPN to learn from the MLLM's own predictions rather than requiring expensive region-level supervision.
How It Works
Q-Zoom operates in a three-stage pipeline. First, the input image is encoded at coarse resolution to extract global context features and generate a query representation. The Dynamic Gating Network then evaluates whether fine-grained perception is necessary for the given query—this is a learned binary decision that compares computational cost against expected accuracy gain. If the gate remains closed (coarse features suffice), inference completes with early exit, avoiding expensive high-resolution processing. If the gate opens, the Self-Distilled Region Proposal Network takes the coarse features and query representation and proposes a sparse set of image regions likely to be relevant to the task. These regions are re-encoded at high resolution and fused with the coarse global features. The final MLLM decoder processes this adaptive multi-resolution representation to generate the output, benefiting from both global context and local fine-grained details only where needed. The self-distillation works by using the MLLM's own predictions on the coarse image as weak supervision signals for training the RPN, eliminating the need for ground-truth region annotations.
Production Impact
For engineers deploying MLLMs in production, Q-Zoom directly addresses the inference latency wall that has prevented high-resolution models from shipping at scale. In a document processing pipeline, you'd replace the current monolithic 'increase resolution everywhere' approach with the adaptive Q-Zoom framework, reducing average inference latency by avoiding unnecessary high-resolution encoding for simple queries while maintaining accuracy on complex visual reasoning tasks. The Dynamic Gating Network acts as a learned cost-benefit analyzer—for queries like 'What is the image about?' it skips high-resolution processing entirely, but for 'Extract all text from this form' it intelligently allocates budget to relevant regions. Integration is relatively clean since Q-Zoom sits between image encoding and the core MLLM; you'd add the gating and RPN modules, retrain the RPN with self-distillation (using existing model predictions as labels), and calibrate the gate's decision threshold based on your latency SLA. The trade-off is modest training overhead (self-distillation and RPN training) and slightly increased model size, but you gain 2-4x inference speedup on typical workloads without accuracy degradation.
Limitations and When Not to Use This
The paper assumes that task-relevant regions can be identified from coarse features and query text alone—this may fail for tasks requiring global spatial context (e.g., scene geometry understanding where relative positions matter). The Dynamic Gating Network's binary decision is coarse-grained; real-world queries often benefit from variable-depth processing depending on content complexity, which this framework doesn't capture. Self-distillation relies on the base MLLM's predictions being accurate—if the model is weak, RPN training will propagate those errors, potentially learning to focus on 'confident mistakes' rather than truly relevant regions. The paper doesn't discuss how Q-Zoom handles out-of-distribution queries or adversarial cases where the gate's learned thresholds may be exploited. Finally, the approach requires retraining the RPN for each new MLLM variant or fine-tuned model, adding operational overhead that may limit adoption in rapidly-iterating production systems.
Research Context
This work builds on two research threads: (1) efficient transformer scaling via sparse attention and token pruning, and (2) vision-language understanding improvements through higher resolution. Q-Zoom synthesizes these by arguing that resolution scaling and sparse processing are complementary—you need adaptive computation, not just static sparsity. It extends prior work on query-aware or task-aware vision processing (e.g., adaptive computation in ViTs) to the MLLM setting where the 'query' is natural language intent, not just visual content. The Region Proposal Network concept borrows from object detection (Faster R-CNN), but applies it to query-driven regions rather than objects. This likely opens new research directions: (1) richer gating mechanisms beyond binary decisions (e.g., continuous depth/resolution control), (2) joint optimization of gating and RPN rather than pipeline training, and (3) extending to multimodal queries that incorporate text, audio, or temporal context to inform region selection.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::