Linking spatial biology and clinical histology via Haiku
| Authors | Yan Cui et al. |
| Year | 2026 |
| HF Upvotes | 1 |
| arXiv | 2605.00925 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Integrating molecular, morphological, and clinical data is essential for basic and translational biomedical research, yet systematic frameworks for jointly modeling these modalities remain limited. Here we present Haiku, a tri-modal contrastive learning model trained on multiplexed immunofluorescence (mIF). It comprises 26.7 million spatial proteomics patches from 3,218 tissue sections across 1,606 patients spanning 11 organ types, with matched hematoxylin and eosin (H&E) histology and clinical metadata aligned in a shared embedding space. Haiku enables three-way cross-modal retrieval, improves downstream classification and clinical prediction tasks over unimodal baselines, and supports zero-shot biomarker inference through fusion retrieval conditioned on clinical metadata-only text descriptions. Across tasks, Haiku outperforms competing approaches, achieving cross-modal retrieval (Recall@50 up to 0.611 versus near-zero baseline), survival prediction (C-index 0.737, +7.91% relative improvement), and zero-shot biomarker inference (mean Pearson correlation 0.718 across 52 biomarkers). Furthermore, we introduce a counterfactual prediction framework in which modifying only clinical metadata while fixing tissue morphology surfaces niche-specific molecular shifts associated with breast cancer stage progression and lung cancer survival outcomes. In a lung adenocarcinoma case study, the counterfactual analysis recovers niche-specific shifts characterized by increased CD8 and granzyme B, reduced PD-L1, and decreased Ki67, broadly consistent with patterns reported for favorable outcomes. We present these counterfactual results as exploratory, hypothesis-generating signals rather than mechanistic claims. These capabilities demonstrate that tri-modal alignment via Haiku enables integrative analysis of spatial biology, bridging molecular measurements with clinical context for biological exploration.
Engineering Breakdown
Plain English
Haiku is a multi-modal AI model that learns from three types of medical data simultaneously: spatial protein imaging (mIF), tissue histology images (H&E), and clinical patient information. The researchers trained it on 26.7 million image patches from over 3,000 tissue samples across 1,606 patients and 11 organ types, teaching the model to find connections between these modalities in a shared mathematical space. The result is a system that can retrieve relevant tissue images when given clinical descriptions, predict patient outcomes better than single-modality approaches, and infer biomarker presence without explicit training labels—essentially building a bridge between what we see under the microscope and patient outcomes in real clinical settings.
Core Technical Contribution
The core novelty is a tri-modal contrastive learning framework that jointly embeds spatial proteomics, histology, and clinical metadata into a unified representation space—something prior work treated as separate problems. Rather than training three independent models, Haiku uses contrastive learning (similar to CLIP) to force related samples from different modalities close together while pushing unrelated ones apart, enabling zero-shot transfer and cross-modal retrieval without task-specific labels. The scale is significant: 26.7 million spatially-resolved protein patches with alignment to clinical outcomes across diverse organs creates a large foundational model for medical imaging that hasn't existed before. The key architectural insight is using clinical metadata as a text conditioning signal for retrieval, allowing queries like "patient age 65, stage 3 cancer" to retrieve relevant tissue regions without seeing that exact sample during training.
How It Works
The model ingests three input streams: (1) spatial multiplexed immunofluorescence images showing protein locations at high resolution, (2) matched H&E histology slides of the same tissue regions, and (3) clinical tabular data (demographics, diagnoses, outcomes) converted to text descriptions. Each modality is encoded through specialized neural network branches—convolutional encoders for images, text encoders for clinical metadata—producing fixed-size embedding vectors. During training, contrastive loss pulls embeddings of the same tissue sample from different modalities close together while pushing embeddings from different samples far apart, similar to how CLIP learns image-text alignment. At inference, you can query with any modality: give it a clinical text description alone and retrieve spatially-resolved protein images that correlate with that clinical profile, or give it a tissue image and retrieve matching clinical cases. The three-way architecture means you get three independent retrieval pathways (spatial→clinical, histology→clinical, clinical→spatial) from a single trained model, enabling flexible downstream use.
Production Impact
This directly solves the fragmentation problem in digital pathology: today, clinicians and researchers work with H&E images, protein maps, and clinical records as separate silos with manual cross-referencing. Haiku enables automated cohort discovery—search for patients with specific clinical profiles and automatically surface relevant tissue regions and biomarkers without manual review, drastically reducing time from hypothesis to validation. For pathology AI pipelines, this becomes a pre-trained foundation model you can fine-tune on disease-specific classification tasks with far less labeled data than training from scratch; the embedding space already captures clinically-relevant tissue patterns. The production trade-offs are real: you need access to multiplexed immunofluorescence data (expensive equipment, ~$500K+ to acquire tissue-level protein maps) and careful alignment between imaging and clinical records, which many institutions lack. Compute cost for inference is moderate (image encoding is standard CNN work), but the data requirements are steep—building equivalent models in-house demands 1000+ well-annotated cases across multiple organs, making this a "buy the pretrained model" scenario for most organizations rather than train-from-scratch.
Limitations and When Not to Use This
The model is trained on retrospective cohorts with existing clinical outcomes, so it learns correlations rather than causation—it can predict which proteins associate with disease stage but not prove those proteins cause disease progression. Generalization to rare diseases, pediatric populations, or non-Western populations is unknown since the paper doesn't break down cohort composition by disease frequency or demography; the 11-organ scope suggests some diversity but likely biased toward common cancers. The zero-shot biomarker inference relies on the assumption that tissue-clinical correlations learned from training data transfer to novel biomarkers, which fails when the biomarker's relationship to clinical outcomes is orthogonal to what the model observed (e.g., a protective mutation with opposite effect direction). Practically, the need for alignment between mIF, H&E, and clinical records means this only works for institutions with integrated digital pathology workflows; most hospitals still operate with siloed systems and analog records, making deployment friction substantial.
Research Context
This builds on the foundation of CLIP-style contrastive learning applied to medical imaging (prior work like BioViL and PathCLIP showed image-text alignment in pathology works), but extends it to include spatial proteomics—a modality that has exploded in the last 3 years with technologies like Visium and MERFISH. The work aligns with the broader trend of building large-scale foundation models for biomedical data (similar to BioBERT for text, AlphaFold for protein structure) that can be adapted downstream rather than trained from scratch per task. It directly addresses a pain point identified in recent reviews on AI for precision medicine: the lack of systems that jointly model molecular, morphological, and clinical data at scale. The research opens a path toward "biomarker discovery as retrieval"—instead of traditional association studies requiring thousands of samples and statistical testing, you could search the embedding space for proteins that co-occur with specific clinical outcomes, potentially accelerating biomarker validation workflows.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::