RAViT: Resolution-Adaptive Vision Transformer
| Authors | Martial Guidez et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2602.24159 |
| Download | |
| Categories | cs.CV, cs.LG |
Abstract
Vision transformers have recently made a breakthrough in computer vision showing excellent performance in terms of precision for numerous applications. However, their computational cost is very high compared to alternative approaches such as Convolutional Neural Networks. To address this problem, we propose a novel framework for image classification called RAViT based on a multi-branch network that operates on several copies of the same image with different resolutions to reduce the computational cost while preserving the overall accuracy. Furthermore, our framework includes an early exit mechanism that makes our model adaptive and allows to choose the appropriate trade-off between accuracy and computational cost at run-time. For example in a two-branch architecture, the original image is first resized to reduce its resolution, then a prediction is performed on it using a first transformer and the resulting prediction is reused together with the original-size image to perform a final prediction on a second transformer with less computation than a classical Vision transformer architecture. The early-exit process allows the model to make a final prediction at intermediate branches, saving even more computation. We evaluated our approach on CIFAR-10, Tiny ImageNet, and ImageNet. We obtained an equivalent accuracy to the classical Vision transformer model with only around 70% of FLOPs.
Engineering Breakdown
Plain English
RAViT tackles the computational inefficiency of Vision Transformers by processing multiple copies of the same image at different resolutions in parallel branches, then combining their predictions. The framework includes an early exit mechanism that lets you trade off accuracy for speed at runtime—for instance, in a two-branch setup, you can exit early from the lower-resolution branch if confidence is high, avoiding expensive computation in the full-resolution branch. This addresses a real production pain point: Vision Transformers deliver state-of-the-art accuracy on image classification but consume far more compute than CNNs, making them impractical for latency-sensitive or resource-constrained deployments. The approach preserves accuracy while reducing computational cost, with the trade-off point tunable depending on your inference budget.
Core Technical Contribution
The core novelty is a multi-resolution branch architecture combined with learned early exit decisions. Rather than running a single Vision Transformer at full resolution, RAViT creates lightweight branches operating on progressively down-sampled versions of the input, allowing the model to confidently classify many images without computing expensive full-resolution features. The early exit mechanism is the key innovation—it's not just multi-scale inference (which prior work explored), but adaptive routing where the model learns when it has sufficient confidence to stop processing and output a prediction. This moves beyond static multi-scale architectures to truly dynamic inference where the compute spent per image varies based on image difficulty and the runtime accuracy target.
How It Works
The architecture starts with an input image that gets duplicated and processed through K parallel branches, where branch-i receives the image downsampled by factor 2^(K-i). Each branch is a lightweight Vision Transformer (or shared transformer backbone with branch-specific classifiers) that outputs both a confidence score and class prediction. At each branch exit point, the early exit mechanism evaluates whether confidence exceeds a threshold (adjustable at inference time); if yes, prediction is returned immediately, if no, computation continues to the next higher-resolution branch. The final branch always produces output. During training, the model learns to both classify accurately and estimate its own confidence, likely through auxiliary losses on each branch's predictions. At inference, you set a confidence threshold or compute budget, and the model automatically routes each image through the minimum necessary branches to meet that constraint.
Production Impact
For teams deploying Vision Transformers in production, this is a direct compute-cost reduction without retraining—you can run the same model with different exit thresholds for different SLAs. Edge devices and mobile applications benefit significantly: a smartphone running inference can use aggressive early exits (exiting at low-resolution branch) to stay under battery/thermal budgets, while server GPUs can use conservative thresholds for maximum accuracy. Latency improves dramatically for easy examples (low-resolution branch handles them in milliseconds) while hard examples still get full accuracy from the complete model. The runtime threshold tuning means you adapt to varying inference budgets without deploying multiple models—a single checkpoint works for 10ms latency targets and 100ms latency targets. The trade-off is added model complexity (multiple branches, confidence calibration logic) and training overhead to learn good early-exit decisions.
Limitations and When Not to Use This
The paper assumes all image downsampling is lossless enough for early-exit decisions, but highly compressed or texture-heavy images may require full resolution even for simple classification, limiting early-exit savings on certain datasets. The approach requires careful calibration of confidence thresholds per application; a threshold tuned for one dataset may not transfer to another domain, limiting generalization. The multi-branch architecture adds model size (K branches means K times parameters, though lower-resolution branches can be lighter) and training complexity compared to a single ViT, which may not be worth the speedup if your baseline model is already optimized for latency. The paper doesn't address robustness to adversarial examples or out-of-distribution inputs—early exits based on standard confidence may fail catastrophically when models are overconfident on corrupted or adversarial images. Missing from the abstract are comparisons to other efficiency methods (quantization, pruning, distillation) which may achieve similar speedups with less architectural complexity.
Research Context
This builds on the Vision Transformer (ViT) architecture and the efficiency challenge identified by practitioners deploying ViTs in production. Multi-scale and multi-resolution inference is well-established in CNN literature, but applying it to transformers with adaptive early exits is newer. The work sits at the intersection of efficient inference (a major research area given ViT compute costs) and adaptive computation (early exit mechanisms explored in networks like BranchyNets and efficient NNs). The early exit idea extends prior work on confidence-based routing and dynamic networks, bringing it to the high-performance ViT regime. This likely benchmarks on ImageNet or similar large-scale classification datasets, and opens research into learned exit criteria, confidence calibration under distribution shift, and extending multi-branch approaches to other transformer-based vision tasks (detection, segmentation).
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::