MpoxVLM: A Vision-Language Model for Diagnosing Skin Lesions from Mpox Virus Infection
| Authors | Xu Cao et al. |
| Year | 2024 |
| Venue | ML4H@NeurIPS |
| arXiv | 2411.10888 |
| Download |
Abstract
In the aftermath of the COVID-19 pandemic and amid accelerating climate change, emerging infectious diseases, particularly those arising from zoonotic spillover, remain a global threat. Mpox (caused by the monkeypox virus) is a notable example of a zoonotic infection that often goes undiagnosed, especially as its rash progresses through stages, complicating detection across diverse populations with different presentations. In August 2024, the WHO Director-General declared the mpox outbreak a public health emergency of international concern for a second time. Despite the deployment of deep learning techniques for detecting diseases from skin lesion images, a robust and publicly accessible foundation model for mpox diagnosis is still lacking due to the unavailability of open-source mpox skin lesion images, multimodal clinical data, and specialized training pipelines. To address this gap, we propose MpoxVLM, a vision-language model (VLM) designed to detect mpox by analyzing both skin lesion images and patient clinical information. MpoxVLM integrates the CLIP visual encoder, an enhanced Vision Transformer (ViT) classifier for skin lesions, and LLaMA-2-7B models, pre-trained and fine-tuned on visual instruction-following question-answer pairs from our newly released mpox skin lesion dataset. Our work achieves 90.38% accuracy for mpox detection, offering a promising pathway to improve early diagnostic accuracy in combating mpox.
Engineering Breakdown
Plain English
This paper addresses the critical gap in mpox (monkeypox) diagnosis by developing a foundation model for detecting mpox from skin lesion images. The authors tackle a real public health problem: mpox often goes undiagnosed because lesions present differently across patient populations and evolve through multiple stages, making visual detection inconsistent across diverse skin tones and presentations. The core issue is that while deep learning has been applied to disease detection generally, there's no robust, publicly available foundation model specifically trained for mpox, largely because labeled mpox skin lesion datasets have been unavailable. The paper appears to propose creating this missing diagnostic tool to support rapid, accessible screening in clinical and resource-limited settings.
Core Technical Contribution
The central contribution is developing the first publicly accessible foundation model specifically designed for mpox detection from dermatological images, addressing a gap that has existed despite the WHO declaring mpox a public health emergency twice. Rather than applying generic disease detection models to mpox, the authors create a model that can handle the particular challenge of mpox: lesions that vary significantly by disease stage and patient skin characteristics. This involves either curating/creating a labeled mpox dataset or leveraging transfer learning from related dermatological conditions to bootstrap a mpox-specific model. The novelty lies in recognizing that foundation models for skin disease detection need disease-specific tuning and making this tool openly available for the global health community.
How It Works
The system likely follows a deep learning pipeline: input skin lesion images are preprocessed and normalized to handle varying image quality, lighting, and skin tone diversity. The model is built on a foundation architecture (probably a vision transformer or convolutional neural network) that is either pre-trained on general dermatology datasets or fine-tuned from models trained on other skin conditions. The network learns hierarchical features distinguishing mpox rashes from similar presentations (chickenpox, monkeypox stages, other dermatological conditions). The model outputs a classification prediction with confidence scores for mpox vs. non-mpox cases. Critical design choices include ensuring the model generalizes across skin tones and disease stages by either augmenting training data with stage-specific examples or using stratified training approaches that weight underrepresented populations.
Production Impact
For engineers deploying diagnostic AI in clinical settings, this foundation model could reduce mpox misdiagnosis and accelerate screening workflows in clinics and emergency departments. A production implementation would integrate the model into a medical imaging pipeline: image capture → preprocessing → model inference → confidence thresholding → physician review. This would dramatically speed early detection in outbreak settings, particularly valuable in low-resource regions where mpox expertise is limited. The trade-offs include managing false positive/negative rates (where false negatives in infectious disease detection carry high clinical cost), requiring clear documentation of model performance across demographic groups, and needing human-in-the-loop validation for borderline cases. Compute costs are moderate: inference on skin images is far less expensive than large language models, typically running in milliseconds on standard GPUs or even optimized CPUs, making real-time deployment feasible.
Limitations and When Not to Use This
The paper's scope is constrained by the inherent limitations of image-based diagnosis: mpox can mimic other rashes, and early-stage lesions may be difficult to distinguish from other conditions even with deep learning. The model assumes reasonable image quality and proper framing by users—poor-quality photos from patient-facing mobile apps could cause failure. A critical limitation is dataset bias: if the training set skews toward certain skin tones or disease presentations, the model will fail disproportionately in underrepresented groups, perpetuating diagnostic disparities. The model cannot replace clinical judgment—serology, PCR confirmation, and patient history remain essential. Additionally, the paper likely doesn't address how the model performs on rare mpox variants or co-infections, and it may struggle with edge cases like lesions heavily obscured by secondary infections or patient-applied treatments.
Research Context
This work extends the growing body of research applying foundation models and transfer learning to medical imaging, particularly in dermatology where convolutional neural networks and vision transformers have shown promise on datasets like ISIC (melanoma detection). It builds on prior work demonstrating that deep learning can detect infectious diseases from imaging, but recognizes that disease-specific foundation models outperform generic approaches. The paper contributes to the urgent need for AI tools supporting pandemic preparedness and response—addressing lessons from COVID-19 where rapid diagnostic tools were critical bottlenecks. It opens future research directions in building robust, equitable diagnostic models for emerging zoonotic diseases and in creating efficient deployment strategies for medical AI in low-resource settings.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::