MuViT: Multi-Resolution Vision Transformers for Learning Across Scales in Microscopy
| Authors | Albert Dominguez Mantes et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2602.24222 |
| Download | |
| Categories | cs.CV, cs.LG |
Abstract
Modern microscopy routinely produces gigapixel images that contain structures across multiple spatial scales, from fine cellular morphology to broader tissue organization. Many analysis tasks require combining these scales, yet most vision models operate at a single resolution or derive multi-scale features from one view, limiting their ability to exploit the inherently multi-resolution nature of microscopy data. We introduce MuViT, a transformer architecture built to fuse true multi-resolution observations from the same underlying image. MuViT embeds all patches into a shared world-coordinate system and extends rotary positional embeddings to these coordinates, enabling attention to integrate wide-field context with high-resolution detail within a single encoder. Across synthetic benchmarks, kidney histopathology, and high-resolution mouse-brain microscopy, MuViT delivers consistent improvements over strong ViT and CNN baselines. Multi-resolution MAE pretraining further produces scale-consistent representations that enhance downstream tasks. These results demonstrate that explicit world-coordinate modelling provides a simple yet powerful mechanism for leveraging multi-resolution information in large-scale microscopy analysis.
Engineering Breakdown
Plain English
MuViT addresses a fundamental problem in microscopy imaging: modern microscopes generate gigapixel images with structures at wildly different scales (fine cellular details to broad tissue organization), but existing vision models only process single-resolution inputs or extract multi-scale features from one view. The authors built a transformer architecture that ingests the same underlying image at multiple resolutions simultaneously and fuses them into a unified representation using a shared world-coordinate system and extended rotary positional embeddings. This allows the model to exploit both wide-field context and high-resolution detail in a single encoder pass, enabling tasks that inherently require multi-scale reasoning without sacrificing either field-of-view or resolution.
Core Technical Contribution
The core innovation is a principled method for multi-resolution fusion inside a transformer encoder that preserves spatial relationships across scales. Rather than concatenating features from separate single-resolution models or using pyramidal Feature extraction from one image, MuViT embeds patches from different resolutions into a unified coordinate system and applies rotary positional embeddings adapted to these coordinates. This enables cross-resolution attention—the model can directly attend from fine-grained patches to coarse ones and vice versa, learning how structures at different scales relate to each other. The technical novelty lies in making positional encodings scale-aware while maintaining a single shared attention mechanism, rather than using separate branches or hierarchical designs.
How It Works
The input is a gigapixel microscopy image that the system samples at multiple resolutions (e.g., 40× and 10× magnification). Each resolution level gets divided into patches (standard ViT-style tokenization), and these patches are embedded into vectors. The key step is coordinate alignment: all patches, regardless of resolution, are mapped to a shared world-coordinate system so that a patch at low resolution and the corresponding high-resolution patches occupying the same physical space have overlapping or adjacent coordinate embeddings. Rotary positional embeddings are extended to work in this multi-resolution coordinate space, encoding both the position within the patch grid and the resolution level. A single transformer encoder then processes all tokens jointly, allowing attention heads to integrate information across resolutions—a token at coarse resolution can directly attend to fine-resolution tokens in its spatial neighborhood. The output is a unified feature representation that captures both scales, which can feed into downstream task heads (classification, segmentation, etc.).
Production Impact
For microscopy image analysis pipelines, this eliminates the need to run separate inference passes at different zoom levels or hand-craft multi-scale feature fusion logic. In practice, you'd replace dual single-resolution ViT models (or inefficient image pyramids) with one MuViT model that processes the image more intelligently—reducing inference latency compared to sequential multi-scale pipelines while improving accuracy by allowing the model to learn cross-scale relationships. The shared coordinate system means you need careful data preprocessing to establish ground-truth scale mappings, and model serving requires loading multiple resolution views into memory (trade-off: more GPU memory during inference vs. simpler architecture and better accuracy). Integration into existing pipelines is straightforward—treat MuViT as a drop-in backbone for downstream tasks like cell type classification, tissue segmentation, or morphology analysis, though you'll need to retrain task heads and ensure your gigapixel slides are preprocessed into the multi-resolution format.
Limitations and When Not to Use This
The paper assumes access to multi-resolution slides of the same scene, which may not always be available in legacy datasets or certain imaging modalities. Computational cost at inference time scales with the total token count across resolutions—processing a gigapixel image at two resolutions simultaneously requires roughly 2× the tokens of a single-resolution baseline, which could be prohibitive for real-time clinical applications without careful batching and caching strategies. The approach likely assumes some regularity in how structures relate across scales (typical for natural tissue), which may not generalize to pathological or unusual morphologies. The paper abstract cuts off mid-sentence and doesn't report actual benchmark numbers or failure case analysis, so production teams would need the full paper to assess real-world performance, latency, and robustness on their specific microscopy hardware and tissue types.
Research Context
This work builds on the transformer-as-backbone paradigm in vision (ViT and follow-ups) and multi-scale feature extraction methods from CNN literature (FPN, U-Net style skip connections), but innovates by doing true concurrent multi-resolution processing in attention rather than sequential or hierarchical fusion. It directly targets medical imaging and microscopy, where gigapixel analysis and multi-scale structure are inherent—related to recent work in computational pathology and digital microscopy that struggles with single-resolution bottlenecks. The extension of rotary positional embeddings to world-coordinate systems opens a research direction for spatially-aware transformers that could apply beyond microscopy to drone imagery, satellite data, or any domain with natural multi-scale structure. Benchmarks would likely include standard histopathology datasets (CAMELYON16/17, or proprietary tissue atlas data) and task-specific metrics, though the abstract doesn't specify which.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::