MARCO: Navigating the Unseen Space of Semantic Correspondence
| Authors | Claudia Cuttano et al. |
| Year | 2026 |
| HF Upvotes | 4 |
| arXiv | 2604.18267 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Recent advances in semantic correspondence rely on dual-encoder architectures, combining DINOv2 with diffusion backbones. While accurate, these billion-parameter models generalize poorly beyond training keypoints, revealing a gap between benchmark performance and real-world usability, where queried points rarely match those seen during training. Building upon DINOv2, we introduce MARCO, a unified model for generalizable correspondence driven by a novel training framework that enhances both fine-grained localization and semantic generalization. By coupling a coarse-to-fine objective that refines spatial precision with a self-distillation framework, which expands sparse supervision beyond annotated regions, our approach transforms a handful of keypoints into dense, semantically coherent correspondences. MARCO sets a new state of the art on SPair-71k, AP-10K, and PF-PASCAL, with gains that amplify at fine-grained localization thresholds (+8.9 [email protected]), strongest generalization to unseen keypoints (+5.1, SPair-U) and categories (+4.7, MP-100), while remaining 3x smaller and 10x faster than diffusion-based approaches. Code is available at https://github.com/visinf/MARCO .
Engineering Breakdown
Plain English
MARCO addresses a critical gap in semantic correspondence models: while dual-encoder architectures using DINOv2 and diffusion backbones achieve strong benchmark performance, they fail dramatically on real-world queries where keypoints differ from training data. The paper introduces MARCO, a unified model that combines a coarse-to-fine spatial refinement objective with self-distillation to expand supervision beyond sparse annotated regions, transforming limited keypoint annotations into dense, semantically coherent correspondence maps. The key insight is that billion-parameter models suffer from poor generalization precisely because they overfit to the exact keypoint locations seen during training rather than learning underlying semantic relationships. By decoupling spatial precision from semantic understanding through their training framework, MARCO achieves both accurate localization and robust generalization to unseen query points.
Core Technical Contribution
The core novelty is a two-stage training framework that explicitly separates the problems of fine-grained localization and semantic generalization. First, a coarse-to-fine objective progressively refines spatial precision, ensuring accurate correspondence at pixel-level granularity. Second, a self-distillation mechanism—trained on sparse keypoint annotations—generates dense pseudo-labels across unannotated regions, effectively bootstrapping semantic understanding from minimal supervision. This differs fundamentally from prior dual-encoder approaches that treat localization and generalization as coupled problems solved simultaneously, often resulting in models that memorize training keypoints rather than learning transferable correspondence patterns. The architectural choice to build upon DINOv2 (which provides strong semantic features) while adding task-specific refinement heads represents a pragmatic balance between leveraging pretrained representations and enabling domain-specific learning.
How It Works
The input is an image pair and sparse keypoint annotations marking semantic correspondences between them. MARCO first extracts multi-scale feature maps from both images using a DINOv2 backbone, which provides semantically rich initial representations. The coarse-to-fine objective then operates hierarchically: at coarse scales, the model learns to identify approximate correspondence regions; at fine scales, it refines predictions to precise pixel coordinates through progressive upsampling and localization refinement. Simultaneously, the self-distillation framework uses the model's predictions on sparsely-annotated keypoints to generate soft targets for unannotated regions, creating dense supervision that teaches semantic consistency across the entire image rather than just at labeled points. The output is a dense correspondence map where every pixel in the query image is assigned a semantic correspondence location in the reference image, along with confidence scores. The two objectives are jointly optimized during training, with the coarse-to-fine path providing spatial supervision and self-distillation expanding the effective supervision signal across orders of magnitude more pixels than annotated.
Production Impact
For teams building correspondence-dependent systems—image stitching, 3D reconstruction, pose estimation, or visual tracking—MARCO solves the generalization problem that currently forces expensive annotation campaigns or domain-specific fine-tuning. In production pipelines, you'd replace dual-encoder inference with a single unified model that handles both coarse and fine correspondence, reducing model count and inference latency. The practical win is handling real-world queries: if your training data has keypoints at faces but your production queries need hand correspondences, MARCO generalizes rather than failing catastrophically. However, real costs include: the self-distillation training loop requires iterative refinement (longer training wall-clock time), you need at least some sparsely-annotated data (not zero-shot), and the model remains computationally heavy (billion-parameter class) so edge deployment requires quantization or distillation. Latency should be reasonable for offline correspondence tasks (image matching, reconstruction preprocessing) but may be tight for real-time tracking at 30+ FPS on embedded hardware.
Limitations and When Not to Use This
MARCO assumes that sparse keypoint annotations are available and representative of semantic correspondences—it cannot perform true zero-shot correspondence on entirely novel object categories not represented in training. The self-distillation mechanism relies on the model's predictions being accurate enough to generate useful pseudo-labels, creating a risk of error propagation where incorrect coarse predictions poison the dense supervision signal. The paper does not address temporal correspondence (video) or handle large deformations/occlusions gracefully, limiting applicability to dynamic scenes or heavily occluded objects. The billion-parameter model size means you cannot deploy MARCO on mobile devices without significant model compression (quantization, distillation), and the paper provides no guidance on whether compression preserves the generalization improvements—a critical gap for production edge deployment. Finally, the evaluation likely uses standard benchmarks (likely YFCC100M or similar) which may not reflect the distribution shift seen in production correspondence tasks, so real-world generalization beyond the benchmark remains empirically unverified.
Research Context
MARCO builds directly on DINOv2, which demonstrated that self-supervised vision transformers learn robust semantic features without task-specific supervision, and extends this foundation for downstream correspondence tasks. The dual-encoder + diffusion backbone architecture was established by recent work in semantic correspondence that showed combining feature extraction with refinement modules improves accuracy, but MARCO innovates by making this architecture generalizable beyond memorized keypoints. The work sits at the intersection of few-shot learning (sparse annotations) and dense prediction (pixel-level correspondence), drawing inspiration from recent advances in self-distillation and pseudo-labeling in semi-supervised vision. This likely advances benchmarks like SPair-71k or HPatches in geometric correspondence, opening a research direction toward 'fine-tuning-free' correspondence models that robustly handle domain shift—a long-standing challenge in practical 3D vision applications.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::