Multimodal Optimal Transport for Unsupervised Temporal Segmentation in Surgical Robotics
| Authors | Omar Mohamed et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2602.24138 |
| Download | |
| Categories | cs.CV, cs.AI |
Abstract
Recognizing surgical phases and steps from video is a fundamental problem in computer-assisted interventions. Recent approaches increasingly rely on large-scale pre-training on thousands of labeled surgical videos, followed by zero-shot transfer to specific procedures. While effective, this strategy incurs substantial computational and data collection costs. In this work, we question whether such heavy pre-training is truly necessary. We propose Text-Augmented Action Segmentation Optimal Transport (TASOT), an unsupervised method for surgical phase and step recognition that extends Action Segmentation Optimal Transport (ASOT) by incorporating textual information generated directly from the videos. TASOT formulates temporal action segmentation as a multimodal optimal transport problem, where the matching cost is defined as a weighted combination of visual and text-based costs. The visual term captures frame-level appearance similarity, while the text term provides complementary semantic cues, and both are jointly regularized through a temporally consistent unbalanced Gromov-Wasserstein formulation. This design enables effective alignment between video frames and surgical actions without surgical-specific pretraining or external web-scale supervision. We evaluate TASOT on multiple benchmark surgical datasets and observe consistent and substantial improvements over existing zero-shot methods, including StrasBypass70 (+23.7), BernBypass70 (+4.5), Cholec80 (+16.5), and AutoLaparo (+19.6). These results demonstrate that fine-grained surgical understanding can be achieved by exploiting information already present in standard visual and textual representations, without resorting to increasingly complex pre-training pipelines. The code will be available at https://github.com/omar8ahmed9/TASOT.
Engineering Breakdown
Plain English
This paper addresses surgical phase recognition in minimally invasive surgery without requiring large labeled datasets. The authors propose TASOT (Text-Augmented Action Segmentation Optimal Transport), which combines video visual features with automatically generated textual descriptions to segment surgical procedures into phases and steps in an unsupervised manner. Instead of pre-training on thousands of labeled surgical videos, TASOT formulates the problem as a multimodal optimal transport problem, leveraging both vision and language modalities to identify temporal boundaries between surgical actions. The key insight is that textual information extracted directly from video can guide temporal segmentation without manual phase labels, reducing the computational and annotation burden traditionally required for surgical video analysis.
Core Technical Contribution
The core novelty is extending Action Segmentation Optimal Transport (ASOT) into a multimodal framework that incorporates automatically generated text descriptions alongside visual features. Rather than treating segmentation as a purely visual problem, TASOT formulates it as an optimal transport problem between multimodal feature distributions, where text acts as an additional signal to disambiguate temporal boundaries between surgical phases. This is technically distinct from prior work because it eliminates the need for supervised phase labels while maintaining the mathematical rigor of optimal transport theory. The contribution is particularly valuable because it shows that unsupervised segmentation is viable in surgical domains through multimodal fusion, challenging the prevailing assumption that large-scale pre-trained models are necessary.
How It Works
TASOT takes as input a surgical video stream and extracts two parallel feature representations: (1) visual features from video frames using a pre-trained encoder, and (2) textual descriptions automatically generated from the video (likely via video-to-text models or caption generation). The method formulates temporal segmentation as finding an optimal transport plan between the visual feature sequence and the textual feature sequence, treating them as probability distributions over time. At each potential temporal boundary, the algorithm computes the cost of transporting visual frames from one phase to another and aligns this with semantic shifts in the generated text descriptions. The optimal transport solution yields a probability map over time indicating phase boundaries, which can be decoded into discrete segments corresponding to different surgical phases and steps. The unsupervised nature means the method learns phase structure directly from the multimodal data without explicit phase labels.
Production Impact
In production surgical AI systems, TASOT would dramatically reduce the annotation burden—surgical teams currently spend thousands of hours labeling phase boundaries in training data. By eliminating the need for large labeled datasets, this approach makes surgical AI deployment feasible for specialized procedures with limited labeled data (rare surgeries, new techniques, hospital-specific variants). The multimodal approach provides interpretability benefits: text descriptions of what's happening serve as intermediate representations that engineers and surgeons can verify, improving transparency compared to black-box visual models. However, practitioners should expect trade-offs: the method likely requires high-quality video-to-text generation (adding a dependency on caption models), may have higher latency due to optimal transport computation per video, and performance probably drops on highly ambiguous phase boundaries where text descriptions alone cannot resolve temporal segmentation. Integration would require building a pipeline that includes text generation, feature extraction, and optimal transport solving, adding engineering complexity compared to standard supervised segmentation networks.
Limitations and When Not to Use This
TASOT's effectiveness is directly coupled to the quality of automatically generated text—if the caption generation model fails to describe key surgical actions accurately, segmentation will degrade, but the paper doesn't clearly characterize this sensitivity. The method assumes sufficient visual-semantic alignment during phases; in surgical videos with long periods of inactivity or purely perceptual changes (small instrument movements without semantic change), the text signal may provide minimal guidance. The unsupervised setting means there's no ground-truth validation until deployment, making it difficult to estimate performance before production use. The paper doesn't address how to set the number of phases/steps when this is unknown a priori, limiting applicability to entirely novel surgical procedures where phase structure hasn't been documented. Additionally, optimal transport computation scales poorly with sequence length, which may create latency bottlenecks for long surgical procedures or real-time applications.
Research Context
This work builds directly on Action Segmentation Optimal Transport (ASOT), extending it from single-modality to multimodal settings—a natural evolution in computer vision research toward leveraging complementary modalities. It contributes to the broader movement questioning the necessity of massive pre-training in specialized domains (surgical robotics), similar to recent work in medical imaging showing that domain-specific unsupervised methods can match supervised baselines. The paper likely benchmarks on surgical video datasets like Cholec80, M2CAI, or similar surgical action recognition benchmarks, which are standard evaluation venues in surgical AI. This opens a research direction toward multimodal unsupervised learning in structured video understanding, potentially influencing how the community approaches other procedural video domains (manufacturing, assembly, repair) where textual descriptions are naturally available or easily generated.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::