Joint-Centric Dual Contrastive Alignment with Structure-Preserving and Information-Balanced Regularization
| Authors | Habibeh Naderi et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2604.16247 |
| Download | |
| Categories | cs.LG, cs.AI |
Abstract
We propose HILBERT (HIerarchical Long-sequence Balanced Embedding with Reciprocal contrastive Training), a cross-attentive multimodal framework for learning document-level audio-text representations from long, segmented sequences in low-resource data settings. HILBERT leverages frozen pre-trained speech and language encoders to extract segment-level features, which are aggregated via cross-modal attention and self-attentive pooling to form modality-specific document representations and a joint cross-attentive embedding. To align modalities while preserving modality-specific structure under severe audio-text dimensional imbalance, we introduce a reciprocal dual contrastive objective that simultaneously aligns audio-to-joint and text-to-joint representations, rather than directly contrasting audio and text alone. Two auxiliary regularizers further stabilize long-sequence fusion: a Centered Kernel Alignment (CKA) loss that preserves structural consistency between each modality and the joint embedding, and a mutual information balancing loss that prevents dominance of a single modality by equalizing information flow from audio and text into the joint space. For downstream prediction, HILBERT employs a Mixture-of-Experts (MoE) classifier over concatenated audio, text, and joint representations to accommodate heterogeneous label regimes. Extensive evaluation across multiple audio-text backbone combinations demonstrates that HILBERT learns semantically meaningful long-sequence representations and achieves superior performance on highly imbalanced multi-class settings.
Engineering Breakdown
Plain English
HILBERT is a multimodal framework that learns joint representations from long audio and text documents in low-resource settings by using frozen pre-trained encoders and a novel dual contrastive loss. The key problem it solves is handling severe dimensional imbalance between audio and text modalities—audio typically produces far higher-dimensional features than text—while maintaining modality-specific structure during alignment. Instead of directly contrasting audio against text, HILBERT introduces a reciprocal dual contrastive objective that aligns both modalities independently to a shared joint embedding space, which reduces the dimensional mismatch problem. The approach segments long sequences into manageable chunks, aggregates them with cross-modal and self-attentive pooling, and uses this architecture to work effectively with limited training data, making it practical for real-world low-resource scenarios.
Core Technical Contribution
The core novelty is the reciprocal dual contrastive training objective, which replaces direct audio-text contrastive learning with a two-stage alignment strategy: audio-to-joint and text-to-joint contrasting done simultaneously. This indirect alignment approach preserves the inherent structure and dimensionality of each modality while still achieving cross-modal coherence, avoiding the forced compression that occurs when directly aligning mismatched feature spaces. The hierarchical architecture—combining cross-modal attention with self-attentive pooling for document-level aggregation—provides a scalable way to handle variable-length, segmented sequences without requiring task-specific fine-tuning of massive pre-trained models. By keeping the base speech and language encoders frozen, the approach reduces compute requirements while remaining effective in low-data regimes, which is a significant practical constraint in many real-world applications.
How It Works
The pipeline begins by segmenting long audio-text document pairs into smaller chunks that fit within the context windows of frozen pre-trained speech and language encoders. Each segment is encoded independently—audio through a speech encoder (producing high-dimensional acoustic embeddings) and text through a language encoder (producing lower-dimensional semantic embeddings)—to extract segment-level features. These segment embeddings are then aggregated using cross-modal attention, where each modality attends to the other to create fused segment representations, followed by self-attentive pooling that learns to weight which segments matter most for the final document representation. This produces three outputs: modality-specific document embeddings (audio-doc and text-doc) and a joint cross-attentive embedding that fuses both modalities. Training uses the reciprocal dual contrastive loss, which pulls audio-doc and text-doc both toward the joint embedding independently, avoiding the dimensional collision problem that would occur if audio and text were contrasted directly. At inference, the joint embedding serves as the document-level multimodal representation for downstream tasks like retrieval or classification.
Production Impact
For production systems handling long-form audio-text data (podcasts, interviews, lectures), HILBERT eliminates the need to fine-tune massive foundation models, reducing compute cost and training time significantly—particularly valuable when data is scarce or labeled examples are expensive. The frozen encoder approach means you can swap in newer or task-specific pre-trained models without retraining, providing modularity and reducing deployment friction in evolving ML pipelines. The hierarchical segmentation strategy naturally handles variable-length inputs, which is critical for real-world documents that don't fit fixed-size windows, and the reciprocal contrastive loss can be adopted into existing multimodal retrieval systems (e.g., cross-lingual search, medical record matching) with minimal architectural changes. Trade-offs include: segment-level encoding loses some global context (you'd need overlapping or sliding windows to mitigate), the method requires pre-trained encoders to be good quality (poor base encoders won't improve without fine-tuning), and the reciprocal objective adds computational overhead during training compared to naive contrastive approaches, though this is offset by not fine-tuning the encoders themselves.
Limitations and When Not to Use This
HILBERT assumes access to quality pre-trained speech and language models, which may not exist for low-resource or non-English languages—if your domain has poor base encoders, the frozen approach becomes a liability rather than an efficiency win. The paper doesn't clearly address how segment boundaries are chosen or what happens at segment boundaries during aggregation, which could degrade representation quality if important audio-text alignments span multiple segments improperly. The reciprocal dual contrastive loss, while solving dimensional imbalance, lacks theoretical justification for why this specific two-stage alignment is superior to other solutions (e.g., adaptive projection layers, modality-specific loss weighting); empirical results would be needed to validate the assumption. Additionally, the approach is evaluated implicitly on document-level tasks but the paper abstract doesn't specify what downstream benchmarks are used, making it unclear how well this translates to standard retrieval, classification, or matching tasks in real-world production scenarios.
Research Context
This work builds on the foundation of multimodal contrastive learning (like CLIP, which aligns vision and text) but innovates specifically for the audio-text modality pair with extreme dimensional imbalance, addressing a gap that most multimodal papers don't tackle. It extends hierarchical attention mechanisms and pooling strategies from document representation learning, combining them with cross-modal fusion in a way that preserves modality-specific semantics—prior work often loses modality-specific information during fusion. The frozen encoder paradigm draws from prompt learning and parameter-efficient fine-tuning trends in LLMs, applying that philosophy to the multimodal representation space and demonstrating that effective alignment doesn't require end-to-end training. This opens research directions into principled methods for aligning modalities with extreme feature space mismatches, better segment aggregation strategies, and evaluation frameworks that measure preservation of modality-specific information during fusion.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::