BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs
| Authors | Nicolas Boizard et al. |
| Year | 2026 |
| HF Upvotes | 32 |
| arXiv | 2604.02045 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Transforming causal generative language models into bidirectional encoders offers a powerful alternative to BERT-style architectures. However, current approaches remain limited: they lack consensus on optimal training objectives, suffer from catastrophic forgetting at scale, and fail to flexibly integrate the vast ecosystem of specialized generative models. In this work, through systematic ablations on the Gemma3 and Qwen3 families, we identify the key factors driving successful adaptation, highlighting the critical role of an often-omitted prior masking phase. To scale this process without original pre-training data, we introduce a dual strategy combining linear weight merging with a lightweight multi-domain data mixture that mitigates catastrophic forgetting. Finally, we augment our encoders by merging them with specialized causal models, seamlessly transferring modality- and domain-specific capabilities. This open-source recipe, designed for any causal decoder LLM, yields BidirLM, a family of five encoders that outperform alternatives on text, vision, and audio representation benchmarks.
Engineering Breakdown
Plain English
This paper addresses a fundamental challenge in modern language models: converting causal generative models (like those used in GPT-style architectures) into bidirectional encoders that can be fine-tuned efficiently without retraining from scratch. The authors systematically study what makes this conversion work by running ablations on Gemma3 and Qwen3 model families, discovering that a prior masking phase is critical—a step that's often overlooked in current approaches. To scale this process without access to original pre-training data, they combine linear weight merging with a lightweight multi-domain data mixture to prevent catastrophic forgetting. The result is a practical framework for repurposing existing generative models as specialized encoders, reducing both the data and compute requirements compared to training encoders from scratch.
Core Technical Contribution
The paper's core innovation is a systematic recipe for converting autoregressive language models into masked bidirectional encoders without catastrophic forgetting or requiring original pre-training data. The key technical breakthrough is identifying and validating the necessity of a prior masking phase—applying masked language modeling objectives before the full encoder transformation—which acts as an essential bridge between the two training paradigms. They introduce a dual weight merging strategy that combines linear interpolation of weights with a carefully curated multi-domain data mixture, allowing the model to retain knowledge from its generative phase while learning new bidirectional representations. This is distinct from prior work because it avoids both the expense of full retraining and the loss of capability that typically occurs when naively fine-tuning generative models into encoders.
How It Works
The system works in three phases: first, take a pre-trained causal generative model (e.g., Gemma3 or Qwen3) and apply a masking phase where you train it with masked language modeling objectives while keeping the causal attention mask structure—this primes the model for bidirectional reasoning without breaking the original weights catastrophically. Second, transition to full bidirectional attention and continue masked language modeling training with a synthetic multi-domain data mixture (constructed without access to original pre-training corpora) designed to balance preservation of generative knowledge with new encoder capabilities. Third, optionally merge the weights of the generative and encoder-phase checkpoints using linear interpolation, weighted toward the encoder phase but anchored by the generative baseline. The masking phase acts as a regularization mechanism that prevents the model from completely abandoning its original learned representations, while the multi-domain mixture provides diverse signal that exercises different parts of the weight space, mitigating forgetting across domains.
Production Impact
For production systems, this work enables a powerful cost reduction: instead of training encoders from scratch (which requires months of compute and access to massive pre-training corpora), teams can now repurpose existing open-source generative models like Gemma3 or Qwen3 into specialized bidirectional encoders in days or weeks. This is particularly valuable for enterprises that have already deployed a generative model and want to add retrieval, classification, or embedding capabilities without maintaining two separate model families. The approach reduces memory footprint during conversion (linear weight merging is cheap) and avoids the need to curate or store original pre-training data. However, the trade-off is that you're constrained to the quality and scale of the base generative model—if Gemma3 has knowledge gaps in a domain, those gaps will persist in the encoder—and the synthetic multi-domain mixture must be carefully designed, which requires domain expertise. Latency is identical to the base model; the conversion process itself (masking phase + fine-tuning) adds 1-4 weeks of training depending on scale, but this is amortized across many downstream tasks.
Limitations and When Not to Use This
The paper relies on systematic ablations on only two model families (Gemma3 and Qwen3), leaving open questions about whether the masking phase recipe generalizes to other architectures like Llama, Mistral, or closed-source models—the mechanism may be architecture-specific. The approach does not address how to prevent catastrophic forgetting in specialized domains where the multi-domain mixture provides insufficient signal; if your target domain is very narrow (e.g., biomedical), the synthetic mixture may still cause drift. The paper's abstract is truncated (ends mid-sentence), suggesting the experimental results and quantitative metrics are missing from this excerpt, which prevents assessment of whether the improvement over baselines is significant enough to justify the added complexity. Finally, the method assumes you have a reasonably good pre-trained generative model to start with; if the base model has fundamental weaknesses, the encoder conversion cannot fix them, and full retraining may still be necessary.
Research Context
This work builds on the long line of research exploring the relationship between generative and discriminative language models, echoing older insights (e.g., the BERT vs. GPT trade-offs) but applied to modern large-scale models where full retraining is prohibitively expensive. It extends recent work on weight merging techniques (like those used in model stitching and ensemble methods) and applies them to a new domain: architectural paradigm conversion. The paper is positioned within the broader trend toward model adaptation and efficiency—alongside LoRA, QLoRA, and prompt tuning—but focuses on a more radical transformation: changing the fundamental training objective and attention structure. It opens up a research direction around 'model conversion as an alternative to retraining,' which could have significant implications for how the ML community thinks about the reusability of large pre-trained models.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::