AnomalyVFM -- Transforming Vision Foundation Models into Zero-Shot Anomaly Detectors
| Authors | Matic Fučka et al. |
| Year | 2026 |
| HF Upvotes | 4 |
| arXiv | 2601.20524 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Zero-shot anomaly detection aims to detect and localise abnormal regions in the image without access to any in-domain training images. While recent approaches leverage vision-language models (VLMs), such as CLIP, to transfer high-level concept knowledge, methods based on purely vision foundation models (VFMs), like DINOv2, have lagged behind in performance. We argue that this gap stems from two practical issues: (i) limited diversity in existing auxiliary anomaly detection datasets and (ii) overly shallow VFM adaptation strategies. To address both challenges, we propose AnomalyVFM, a general and effective framework that turns any pretrained VFM into a strong zero-shot anomaly detector. Our approach combines a robust three-stage synthetic dataset generation scheme with a parameter-efficient adaptation mechanism, utilising low-rank feature adapters and a confidence-weighted pixel loss. Together, these components enable modern VFMs to substantially outperform current state-of-the-art methods. More specifically, with RADIO as a backbone, AnomalyVFM achieves an average image-level AUROC of 94.1% across 9 diverse datasets, surpassing previous methods by significant 3.3 percentage points. Project Page: https://maticfuc.github.io/anomaly_vfm/
Engineering Breakdown
Plain English
This paper tackles zero-shot anomaly detection—finding abnormal regions in images without any training examples from the target domain. The authors show that vision foundation models (VFMs) like DINOv2 have underperformed compared to vision-language models like CLIP, and they identify two root causes: insufficient diversity in existing anomaly datasets and shallow adaptation strategies. They propose AnomalyVFM, a framework that combines a three-stage synthetic dataset generation scheme with parameter-efficient adaptation to transform any pretrained VFM into a strong anomaly detector. This approach bridges the performance gap and makes VFMs competitive with VLM-based methods for zero-shot anomaly detection.
Core Technical Contribution
The core innovation is decomposing the zero-shot anomaly detection problem into two addressable sub-problems: dataset diversity and model adaptation depth. Rather than treating VFMs as fixed feature extractors, the authors propose a systematic synthetic data generation pipeline (three-stage scheme) combined with parameter-efficient fine-tuning that preserves the foundational knowledge while adapting to anomaly detection specifics. This is a departure from prior VFM work that either used shallow adaptation or relied entirely on language models for semantic grounding. The contribution is both methodological (the synthetic generation framework) and architectural (the adapted VFM layers that remain efficient).
How It Works
The framework operates in three stages: (1) synthetic anomaly dataset generation, which creates diverse in-domain anomalies without access to real examples, improving the data foundation for adaptation; (2) parameter-efficient adaptation of the pretrained VFM, where only a subset of weights or adapter layers are fine-tuned rather than the full model, reducing computational overhead while maintaining transfer learning benefits; (3) inference-time anomaly detection where the adapted VFM produces dense feature maps that are processed to identify and localize abnormal regions. The input is an unlabeled test image and a natural language description of what is normal; the VFM backbone extracts multi-scale features; these features are compared against learned normality patterns; the output is an anomaly map highlighting suspicious regions. The key insight is that synthetic data diversity combined with focused adaptation allows VFMs to learn domain-specific anomaly patterns without catastrophic forgetting of their foundational knowledge.
Production Impact
For engineers building anomaly detection systems, this approach eliminates a major blocker: the need for large, labeled in-domain datasets. You can deploy a detector for a new visual domain (industrial inspection, medical imaging, security) with minimal data collection—just a description of normalcy and the framework handles synthetic augmentation. In production pipelines, you'd replace image classification or segmentation models with this VFM-based detector, getting interpretable anomaly localization heatmaps at minimal additional cost. Compute overhead is modest since parameter-efficient adaptation (LoRA-style or adapter modules) keeps training memory and latency low—suitable for edge deployment or frequent retraining. The trade-off is that synthetic data may not capture all real-world anomaly modes, and performance degrades gracefully with domain shift rather than catastrophically, requiring validation thresholds tuned per deployment.
Limitations and When Not to Use This
The paper's reliance on synthetic data generation assumes that artificially created anomalies are representative of real failures—this may not hold for subtle, context-dependent anomalies in high-stakes domains like medical diagnosis. The approach assumes a clear definition of 'normal' can be articulated linguistically or demonstrated, which breaks down in domains where normality itself is ambiguous or varies by context. There's likely a performance ceiling imposed by the VFM's inherent feature quality; if the foundation model wasn't trained on diverse enough visual data, no adaptation strategy will fully compensate. The paper doesn't deeply address handling extremely rare anomalies or adversarial perturbations, and computational cost of synthetic data generation at scale isn't thoroughly analyzed. Finally, the evaluation is likely limited to standard benchmarks (MVTec-AD, similar); real-world performance on proprietary or highly specialized visual domains remains unclear.
Research Context
This work directly challenges the implicit assumption in recent anomaly detection literature that vision-language models (CLIP-based) are inherently superior to pure vision models for zero-shot tasks. It builds on a growing trend of leveraging foundation models (DINO, DINOv2, CLIP) for downstream tasks with minimal finetuning, particularly in the unsupervised/zero-shot regime. The paper sits in the intersection of few-shot/zero-shot learning and anomaly detection—two areas that have traditionally been separate but are converging as foundation models mature. It likely advances benchmarks on datasets like MVTec-AD or VisA, and opens a research direction: can we systematically improve any vision foundation model for task-specific detection with smart synthetic data and lightweight adaptation? Future work will likely explore whether similar strategies apply to other dense prediction tasks (segmentation, depth) or multi-modal anomaly detection.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::