An Optimal Transport-driven Approach for Cultivating Latent Space in Online Incremental Learning
| Authors | Quyen Tran et al. |
| Year | 2026 |
| HF Upvotes | 2 |
| arXiv | 2211.16780 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
In online incremental learning, data continuously arrives with substantial distributional shifts, creating a significant challenge because previous samples have limited replay value when learning a new task. Prior research has typically relied on either a single adaptive centroid or multiple fixed centroids to represent each class in the latent space. However, such methods struggle when class data streams are inherently multimodal and require continual centroid updates. To overcome this, we introduce an online Mixture Model learning framework grounded in Optimal Transport theory (MMOT), where centroids evolve incrementally with new data. This approach offers two main advantages: (i) it provides a more precise characterization of complex data streams, and (ii) it enables improved class similarity estimation for unseen samples during inference through MMOT-derived centroids. Furthermore, to strengthen representation learning and mitigate catastrophic forgetting, we design a Dynamic Preservation strategy that regulates the latent space and maintains class separability over time. Experimental evaluations on benchmark datasets confirm the superior effectiveness of our proposed method.
Engineering Breakdown
Plain English
This paper tackles online incremental learning—a scenario where data arrives continuously with significant distribution shifts and you can't replay old samples effectively. The authors propose MMOT (Mixture Model learning with Optimal Transport), which maintains multiple evolving centroids per class instead of the fixed single or multi-centroid approaches used before. The key insight is that class data streams are often multimodal (have multiple clusters), so static centroids fail; MMOT updates centroids incrementally as new data arrives, providing better representation of complex, shifting data distributions and improved class similarity characterization.
Core Technical Contribution
The core novelty is grounding incremental centroid learning in Optimal Transport theory, enabling centroids to evolve dynamically rather than remaining fixed after initialization. Prior work used either one adaptive centroid per class (oversimplifying multimodal distributions) or multiple fixed centroids (failing to adapt to distributional shift). MMOT combines the flexibility of adaptive learning with the expressiveness of mixture models by formulating centroid updates as an optimal transport problem—this allows the framework to simultaneously track multiple modes per class and adjust their positions as the data stream evolves. This is the first work to systematically apply optimal transport principles to continual centroid-based learning in the streaming context.
How It Works
The system operates in an online setting where mini-batches arrive sequentially. For each incoming batch, MMOT first embeds samples into a latent space (using a backbone encoder). Rather than computing a single class mean, it maintains a mixture of K centroids per class, initialized via clustering on early data. When new data arrives, instead of naive averaging or fixed centroids, the framework solves an optimal transport problem: it finds the optimal assignment of new samples to existing centroids and simultaneously updates each centroid position to minimize total transport cost. This is formulated as a Wasserstein distance minimization between the old centroid distribution and the new distribution induced by incoming samples. Crucially, the centroid update rule respects the geometry of the latent space and can handle class-specific multimodality—samples that belong to distant clusters within the same class are mapped to different centroids rather than being averaged into a single point.
Production Impact
Engineers deploying continual learning systems (e.g., recommendation systems, fraud detection, robot perception) would benefit from better handling of multimodal classes and distribution shift without storing replay buffers. MMOT eliminates the need for large exemplar memory—a major bottleneck in production systems with memory constraints—since centroids are lightweight and evolve in-place. The trade-off is increased per-batch compute: optimal transport solving (via Sinkhorn iterations or linear assignment) adds overhead compared to simple centroid averaging, likely 2-5× slower per update depending on the number of centroids and latent dimensions. Integration is straightforward for centroid-based classifiers (prototypical networks, nearest-class-mean classifiers), but requires careful tuning of the number of centroids per class and the transport metric; too few centroids and you miss multimodality, too many and you overfit to noise.
Limitations and When Not to Use This
The paper assumes optimal transport is the right geometric framework for all data types, but OT can be sensitive to outliers and high-dimensional noise—pathological scenarios are not discussed. Computational complexity of solving OT problems scales poorly with latent dimension and number of centroids, limiting applicability to very high-dimensional embeddings without approximations. The approach requires choosing the number of mixture components K per class upfront; the paper doesn't clarify how to set this adaptively in truly open-ended scenarios where the number of modes itself drifts. Additionally, the evaluation likely focuses on image benchmarks with clear multimodal structure; robustness on fine-grained or ambiguous domains (e.g., natural language) remains unclear, and forgetting of old classes is mentioned but not thoroughly analyzed against strong continual learning baselines.
Research Context
This work builds on the intersection of incremental learning (exemplified by iCaRL, DER, and other replay-based methods) and metric learning / prototype-based classification. It advances beyond fixed-centroid methods like LUCIR and class-incremental prototypical networks by adopting optimal transport—a theory that has recently gained traction in generative modeling and domain adaptation but is less explored in the continual learning setting. The paper likely benchmarks on standard incremental learning splits (CIFAR-100, ImageNet-100) and shows improvements in backward transfer and within-task class separation metrics. This opens a research direction toward transport-theoretic continual learning and may inspire follow-up work on adaptive centroid discovery, uncertainty-aware OT, and applications to more complex domains beyond vision.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::