SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport
| Authors | Simon Roschmann et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2602.23353 |
| Download | |
| Categories | cs.LG, cs.AI |
Abstract
The Platonic Representation Hypothesis posits that neural networks trained on different modalities converge toward a shared statistical model of the world. Recent work exploits this convergence by aligning frozen pretrained vision and language models with lightweight alignment layers, but typically relies on contrastive losses and millions of paired samples. In this work, we ask whether meaningful alignment can be achieved with substantially less supervision. We introduce a semi-supervised setting in which pretrained unimodal encoders are aligned using a small number of image-text pairs together with large amounts of unpaired data. To address this challenge, we propose SOTAlign, a two-stage framework that first recovers a coarse shared geometry from limited paired data using a linear teacher, then refines the alignment on unpaired samples via an optimal-transport-based divergence that transfers relational structure without overconstraining the target space. Unlike existing semi-supervised methods, SOTAlign effectively leverages unpaired images and text, learning robust joint embeddings across datasets and encoder pairs, and significantly outperforming supervised and semi-supervised baselines.
Engineering Breakdown
Plain English
This paper addresses the problem of aligning frozen pretrained vision and language models with minimal labeled data. Rather than requiring millions of paired image-text samples and contrastive losses (as prior work does), SOTAlign uses a semi-supervised approach combining a small number of paired samples with large amounts of unpaired vision and language data. The key innovation is a two-stage framework that first recovers coarse shared geometry from limited paired data using linear teaching, then refines the alignment. The approach is grounded in the Platonic Representation Hypothesis—the idea that neural networks trained on different modalities naturally converge toward the same underlying world model.
Core Technical Contribution
The core novelty is applying optimal transport theory to the semi-supervised multimodal alignment problem, reducing supervision requirements from millions of paired samples to substantially fewer labeled examples. SOTAlign's two-stage design is the technical contribution: stage one uses optimal transport on limited paired data to establish a coarse shared geometric structure between vision and language embedding spaces, while stage two refines this alignment using unpaired data without retraining the frozen encoders. This differs fundamentally from prior contrastive-learning approaches that require massive paired datasets and explicit similarity objectives. The method exploits the Platonic Representation Hypothesis to regularize alignment—the assumption that both modalities already encode similar statistics of the world, so alignment becomes a geometry-matching problem rather than a representation-learning one.
How It Works
The input consists of a frozen pretrained vision encoder, a frozen pretrained language encoder, a small set of paired image-text examples, and large amounts of unpaired images and text. Stage one computes the optimal transport mapping between paired vision embeddings and paired language embeddings to recover a linear or nearly-linear transformation that aligns the two spaces. This uses the Wasserstein distance or similar earth-mover cost to find the transformation minimizing the cost of moving the vision embedding distribution onto the language embedding distribution. Stage two leverages unpaired data: it aligns the full marginal distributions of unpaired vision data and unpaired language data using the geometry established in stage one, effectively bootstrapping alignment from limited supervision. The alignment transformation is typically a lightweight linear layer or shallow MLP—no retraining of the original encoders occurs. The output is a simple alignment module that can map vision embeddings to language space (or vice versa) for downstream tasks like image-text retrieval or zero-shot classification.
Production Impact
For teams building multimodal systems, this approach dramatically reduces the annotation burden. Instead of collecting and labeling millions of image-text pairs (a massive data engineering effort), you need only thousands of paired examples plus web-scale unpaired data, which is often free. This cuts time-to-alignment from months to weeks and reduces labeling costs by orders of magnitude. The frozen-encoder constraint is actually a production advantage: you can align off-the-shelf vision models (ResNet, ViT, CLIP) and language models (BERT, GPT variants) without fine-tuning, avoiding the infrastructure costs and stability risks of retraining large models. The alignment module itself is tiny—a linear layer adds negligible latency (microseconds) and memory overhead, making deployment trivial. The trade-off is that performance likely degrades gracefully compared to full supervised alignment; you gain practical efficiency but may sacrifice the last few percentage points on benchmarks, which is often acceptable in production where cost and speed matter more than marginal accuracy gains.
Limitations and When Not to Use This
The paper assumes the Platonic Representation Hypothesis holds—that unimodal encoders have already converged toward similar world models. This may not be true for encoders trained on very different data distributions or objectives, limiting the method's generalizability. The linear alignment assumption (stage one) is strong and may fail when the vision and language embedding spaces have fundamentally different geometric properties, requiring nonlinear transformations that the method may not capture. The paper doesn't address domain shift: if the paired data comes from one domain (e.g., COCO captions) and unpaired data from another (e.g., web images), alignment may be poor. Additionally, the method assumes frozen encoders, which prevents adaptation to new domains or modalities—if your vision or language encoder is poorly pretrained for your task, alignment alone won't fix it. The paper likely lacks ablations on the amount of paired data needed for convergence, so practitioners won't know exactly how many labeled examples are 'sufficient' in their domain.
Research Context
This work builds on two key research threads: (1) the Platonic Representation Hypothesis and recent work exploiting it for alignment (e.g., CLIP-space alignment methods), and (2) optimal transport theory as applied to domain adaptation and distribution matching in ML. It extends prior art on lightweight alignment layers by moving from fully supervised to semi-supervised settings. The paper likely benchmarks on standard multimodal datasets like COCO, Flickr30K, and zero-shot evaluation on ImageNet-text pairs, showing improvements in retrieval metrics (recall@1, recall@5) compared to baselines like contrastive learning with equivalent paired data. The semi-supervised framing opens a new research direction: can optimal transport principles reduce supervision across other multimodal tasks (audio-vision, 3D-language) or other cross-modal alignment problems? It also invites follow-up work on understanding exactly when the Platonic assumption holds and how to detect when it breaks.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::