OmniJigsaw: Enhancing Omni-Modal Reasoning via Modality-Orchestrated Reordering
| Authors | Yiduo Jia et al. |
| Year | 2026 |
| HF Upvotes | 23 |
| arXiv | 2604.08209 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
To extend the reinforcement learning post-training paradigm to omni-modal models for concurrently bolstering video-audio understanding and collaborative reasoning, we propose OmniJigsaw, a generic self-supervised framework built upon a temporal reordering proxy task. Centered on the chronological reconstruction of shuffled audio-visual clips, this paradigm strategically orchestrates visual and auditory signals to compel cross-modal integration through three distinct strategies: Joint Modality Integration, Sample-level Modality Selection, and Clip-level Modality Masking. Recognizing that the efficacy of such proxy tasks is fundamentally tied to puzzle quality, we design a two-stage coarse-to-fine data filtering pipeline, which facilitates the efficient adaptation of OmniJigsaw to massive unannotated omni-modal data. Our analysis reveals a ``bi-modal shortcut phenomenon'' in joint modality integration and demonstrates that fine-grained clip-level modality masking mitigates this issue while outperforming sample-level modality selection. Extensive evaluations on 15 benchmarks show substantial gains in video, audio, and collaborative reasoning, validating OmniJigsaw as a scalable paradigm for self-supervised omni-modal learning.
Engineering Breakdown
Plain English
OmniJigsaw is a self-supervised learning framework that trains multi-modal AI models to better understand video and audio together by asking them to reconstruct the correct temporal order of shuffled audio-visual clips. The authors use three complementary strategies—Joint Modality Integration, Sample-level Modality Selection, and Clip-level Modality Masking—to force the model to learn meaningful cross-modal representations rather than relying on a single modality. A critical innovation is their two-stage data filtering pipeline (coarse-to-fine) that ensures only high-quality audio-visual pairs are used for training, which dramatically improves the efficiency of the self-supervised proxy task. This work extends reinforcement learning post-training methods from text-only models to truly multi-modal systems that can reason across video and audio simultaneously.
Core Technical Contribution
The core novelty is reframing temporal sequence reconstruction as a principled self-supervised learning objective for multi-modal models, where shuffled clip reordering becomes the proxy task that drives cross-modal integration. Unlike prior work that treats modalities independently or uses contrastive losses, OmniJigsaw forces tight coupling between audio and video through three orthogonal modality selection strategies that operate at different granularities (joint, sample-level, and clip-level). The two-stage data filtering pipeline is technically novel because it addresses a fundamental gap in self-supervised multi-modal learning: recognizing that proxy task quality directly determines representation quality, and therefore designing an explicit coarse-to-fine filtering mechanism to scale to massive unlabeled datasets without degradation. This moves beyond treating all unlabeled multi-modal data as equally valuable and instead establishes that selective, high-quality data curation is a core component of the framework.
How It Works
The input is large-scale unlabeled video-audio pairs that are first passed through a coarse filtering stage to remove obviously misaligned or low-quality clips (e.g., silent videos, extreme duration mismatches). In the fine filtering stage, a lightweight model scores remaining clip pairs to identify those with strong audio-visual correlation. The core training mechanism then shuffles the temporal order of audio-visual clip sequences and trains the model to predict the correct chronological reconstruction—a self-supervised objective that requires understanding both visual and temporal context from video frames and audio spectrograms. The three modality integration strategies operate as follows: Joint Modality Integration processes audio and video features together from the start; Sample-level Modality Selection allows the model to selectively attend to the more informative modality for each training sample; and Clip-level Modality Masking randomly masks either the audio or video track during training to force the model to learn robust representations even with missing information. The model learns by minimizing the error in predicting correct temporal orderings, which implicitly drives it to extract features that capture semantic relationships across modalities rather than shallow temporal patterns.
Production Impact
For teams building video understanding systems, OmniJigsaw eliminates the need for expensive labeled video-audio datasets by leveraging massive unlabeled video repositories—a major cost reduction since video annotation is one of the most expensive data modalities in production ML. The framework directly improves downstream task performance on video-audio understanding problems (e.g., video captioning, audio-visual event detection, cross-modal retrieval) because pre-trained representations are inherently multi-modal, reducing the engineering complexity of late-fusion architectures. In production pipelines, you would replace standard single-modality pre-training with OmniJigsaw's pre-training phase, then fine-tune on your specific task—this adds one additional data filtering stage upfront but saves significant annotation effort downstream. The trade-offs are non-trivial: the coarse-to-fine filtering pipeline adds wall-clock time to data preparation (potentially 10-20% overhead), the model checkpoint size is larger due to dual-modality encoders, and inference latency increases slightly because both audio and video must be processed simultaneously rather than independently.
Limitations and When Not to Use This
The paper assumes that temporal coherence in shuffled clips is a sufficient proxy for learning meaningful multi-modal representations, but this may not hold when audio and video are weakly correlated or out of sync in the source data (e.g., videos with voiceovers, music, or ambient sound where the audio doesn't directly reflect visual content). The coarse-to-fine filtering pipeline requires careful hyperparameter tuning to the specific dataset and video domain, and the authors do not provide clear guidelines on how to set these thresholds when adapting to new domains—this could become a bottleneck in production. The framework is evaluated only on video-audio pairs and may not generalize to truly omni-modal scenarios involving text, 3D pose, or sensor data, limiting its claim as a general omni-modal approach. Additionally, the paper does not address failure modes when audio and video are intentionally misaligned (e.g., deepfakes, edited content) or when the temporal signal is ambiguous, which are important robustness concerns for deployment.
Research Context
OmniJigsaw builds on the recent success of self-supervised learning in vision (e.g., masked image modeling, contrastive learning) and extends this paradigm to the multi-modal setting by leveraging temporal structure—a direction that has seen growing interest as video understanding becomes more central to real-world AI systems. The work is situated within the broader trend of moving away from contrastive objectives (which have dominated multi-modal pre-training) toward reconstructive objectives that capture richer structural information, similar to trends in language modeling where next-token prediction outperformed earlier contrastive approaches. It directly complements reinforcement learning post-training paradigms by showing how to create high-quality self-supervised representations that can later be refined with RL objectives, opening a research direction where RL is applied to multi-modal models rather than only text models. The two-stage data filtering contribution is likely to influence future work on scaling self-supervised learning to noisy, massive unlabeled datasets by establishing data quality as a first-class design concern alongside model architecture.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::