PRIMA: Pre-training with Risk-integrated Image-Metadata Alignment for Medical Diagnosis via LLM
| Authors | Yiqing Wang et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2602.23297 |
| Download | |
| Categories | cs.CV |
Abstract
Medical diagnosis requires the effective synthesis of visual manifestations and clinical metadata. However, existing methods often treat metadata as isolated tags, failing to exploit the rich semantic knowledge embedded in clinical descriptions. We propose PRIMA (Pre-training with Risk-integrated Image-Metadata Alignment), a framework that integrates domain-specific knowledge into multi-modal representation learning. We first curate an expert corpus of risk-disease correlations via Retrieval-Augmented Generation (RAG) to refine Clinical ModernBERT, embedding diagnostic priors into the text encoder. To bridge the modality gap, we introduce a dual-encoder pre-training strategy utilizing DINOv3 and our refined BERT, optimized by a suite of four complementary loss functions. These losses are designed to capture multi-granular semantic alignment and handle the ambiguity of clinical correlations through soft labels. Finally, we leverage Qwen-3 to fuse these aligned features for precise disease classification. Extensive experiments demonstrate that PRIMA effectively harmonizes pixel-level features with abstract clinical expertise, significantly outperforming other state-of-the-art methods. Notably, our framework achieves superior robustness without the need for massive data collection or exhaustive computational resources. Our code will be made public upon acceptance.
Engineering Breakdown
Plain English
PRIMA is a framework for medical diagnosis that combines visual images with clinical metadata by pre-training two synchronized encoders—one for images (DINOv3) and one for clinical text (a refined Clinical ModernBERT). The key innovation is enriching the text encoder with domain-specific knowledge about risk-disease correlations using Retrieval-Augmented Generation (RAG) before training. The paper proposes a dual-encoder architecture optimized by four complementary loss functions to align image and metadata representations into a shared space, enabling more accurate medical diagnosis than methods that treat metadata as isolated tags. This approach directly addresses the gap in existing systems that fail to leverage the rich semantic knowledge embedded in clinical descriptions.
Core Technical Contribution
The core novelty lies in the risk-integrated pre-training pipeline that embeds diagnostic priors into medical language models before cross-modal alignment. Rather than treating clinical metadata as unstructured tags, the authors curate an expert corpus of risk-disease correlations via RAG, which refines Clinical ModernBERT to encode domain-specific diagnostic knowledge. This refined text encoder is then paired with DINOv3 in a dual-encoder pre-training strategy, where a suite of four complementary loss functions jointly optimize image-text alignment while respecting the semantic structure of medical knowledge. The innovation is fundamentally about injecting expert domain knowledge earlier in the pipeline—at the language model level—rather than attempting to recover it during multi-modal training alone.
How It Works
The system operates in three stages: first, an RAG-based pipeline retrieves and integrates expert knowledge about risk-disease correlations to fine-tune Clinical ModernBERT, creating a text encoder that understands diagnostic priors; second, this refined BERT encoder is paired with DINOv3 (an image encoder pre-trained on general vision tasks) in a dual-encoder architecture; third, during pre-training, both encoders process aligned image-metadata pairs, with four complementary loss functions optimizing their joint representation space. The input is a medical image paired with clinical metadata (patient history, symptoms, lab values). The image flows through DINOv3 to produce visual embeddings, while the clinical text flows through the RAG-enhanced BERT to produce semantic embeddings. The four loss functions (likely including contrastive, matching, and alignment objectives—though the abstract cuts off before specifying them) guide both encoders toward a shared embedding space where semantically similar medical concepts align regardless of modality. The output is a joint representation model that can support downstream diagnostic tasks by leveraging both visual and textual medical signals.
Production Impact
For production medical AI systems, PRIMA directly addresses a critical gap: most current pipelines either ignore clinical metadata or treat it as flat categorical features, discarding valuable semantic information from patient history and clinical notes. By embedding domain knowledge into the language encoder before cross-modal training, this approach should improve diagnostic accuracy and reduce false positives/negatives compared to image-only or metadata-only systems. Implementation would require: (1) curating or licensing an expert corpus of risk-disease correlations (data engineering cost), (2) re-training or fine-tuning Clinical ModernBERT with RAG (1-2 weeks on modern GPUs depending on corpus size), (3) integrating DINOv3 + refined BERT into inference pipelines (modest added latency for dual-encoder inference, ~50-200ms per case depending on hardware). The trade-off is increased model complexity—you now maintain two large encoders instead of one—and dependency on high-quality clinical metadata, which may be inconsistent across hospital systems. Adoption would most impact departments with access to both imaging and structured/unstructured clinical notes (radiology, oncology, pathology).
Limitations and When Not to Use This
The paper assumes clinical metadata is available, well-structured, and semantically rich—assumptions that rarely hold across healthcare systems with varying EHR standards and documentation practices. The approach requires expert-curated risk-disease correlations via RAG, creating a dependency on the quality and completeness of that corpus; rare diseases or novel presentations not well-represented in the expert knowledge base may be misdiagnosed. The abstract cuts off before revealing the four loss functions, which are critical for understanding whether the approach handles class imbalance (common in medical datasets) or failure modes when image and metadata are contradictory. It's unclear how the method generalizes across institutions with different patient populations, imaging protocols, or clinical documentation styles—these domain-shift challenges are common in medical AI but not addressed in the abstract. Follow-up work should establish privacy guarantees for the RAG corpus (clinical data is sensitive), measure robustness to missing or noisy metadata, and validate on truly independent external test sets beyond single-institution benchmarks.
Research Context
PRIMA builds on the multi-modal representation learning trend (CLIP, BLIP, LLaVA) but introduces domain-specific innovation by pre-enriching one modality with expert knowledge before alignment—a strategy less common in general-purpose vision-language models. It directly addresses known limitations of prior medical multi-modal work: MedFuseNet, RadiologyGPT, and other systems treat clinical data as auxiliary rather than semantically rich, whereas PRIMA prioritizes metadata semantics via RAG integration. The paper advances the intersection of medical AI, retrieval-augmented generation, and contrastive learning, extending recent work on instruction-tuned medical LLMs (e.g., MedAlpaca, LLaMA-Med) into the multi-modal diagnosis domain. By combining DINOv3 (a recent self-supervised vision model) with knowledge-enriched BERT, the work opens a research direction: can domain-specific knowledge injection at the encoder level outperform post-hoc alignment methods on specialized tasks like medical diagnosis, and how does this scale to other high-stakes domains (law, finance)?
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::