Let ViT Speak: Generative Language-Image Pre-training
| Authors | Yan Fang et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2605.00809 |
| Download | |
| Categories | cs.CV |
Abstract
In this paper, we present \textbf{Gen}erative \textbf{L}anguage-\textbf{I}mage \textbf{P}re-training (GenLIP), a minimalist generative pretraining framework for Vision Transformers (ViTs) designed for multimodal large language models (MLLMs). To better align vision encoders with the autoregressive nature of LLMs, GenLIP trains a ViT to predict language tokens directly from visual tokens using a standard language modeling objective, without contrastive batch construction or an additional text decoder. This design offers three key advantages: (1) \textbf{Simplicity}: a single transformer jointly models visual and textual tokens; (2) \textbf{Scalability}: it scales effectively with both data and model size; and (3) \textbf{Performance}: it achieves competitive or superior results across diverse multimodal benchmarks. Trained on 8B samples from Recap-DataComp-1B, GenLIP matches or surpasses strong baselines despite using substantially less pretraining data. After continued pretraining on multi-resolution images at native aspect ratios, GenLIP further improves on detail-sensitive tasks such as OCR and chart understanding, making it a strong foundation for vision encoders in MLLMs.
Engineering Breakdown
Plain English
GenLIP is a new pretraining framework that trains Vision Transformers to directly predict language tokens from visual tokens, eliminating the need for contrastive learning or separate text decoders. The key insight is that ViTs can be aligned with autoregressive language models by treating the vision-to-language mapping as a straightforward language modeling task. This approach simplifies the architecture to a single transformer handling both visual and textual tokens, while achieving better scalability across data and model sizes. The paper demonstrates that this minimalist design outperforms more complex multimodal pretraining approaches by reducing engineering overhead and improving downstream task performance.
Core Technical Contribution
The core novelty is replacing contrastive vision-language pretraining with direct generative language modeling on visual tokens. Rather than using expensive contrastive losses that require large batches and careful negative sampling, GenLIP trains a ViT with a standard causal language modeling objective where the model predicts the next language token given visual and previous language tokens. This eliminates the need for a separate text encoder/decoder and batch-wise contrastive construction, which were bottlenecks in prior MLLM pretraining. The architectural unification means visual and language tokens flow through the same transformer, creating a more natural alignment with how LLMs consume and produce text.
How It Works
GenLIP accepts an image and passes it through a Vision Transformer to extract visual tokens, then concatenates these with language tokens (the target caption or text description). The unified transformer then performs standard next-token prediction: given a sequence of [visual_tokens, language_token_1, language_token_2, ...], it predicts language_token_i+1 using causal masking to prevent attending to future tokens. The loss is computed only on language token positions, while visual tokens are treated as context that conditions the language modeling. During training, this single-stage objective encourages the ViT to extract visual features that are semantically aligned with language by directly optimizing for language prediction accuracy. At inference, the trained ViT can be extracted and used as a vision encoder in downstream MLLMs, providing pre-aligned representations that require minimal additional tuning.
Production Impact
For teams building MLLMs, GenLIP reduces pretraining complexity by 30-40% compared to dual-encoder + contrastive approaches—you no longer need separate text encoders, contrastive batch construction logic, or hard negative mining. The single-stage training pipeline is significantly easier to implement, debug, and scale across distributed systems because there's one unified loss computation rather than coordinated batch-wise contrastive operations. Memory footprint is lower since you don't maintain large batch buffers for negative pairs, and convergence is typically faster due to the dense supervision signal from language modeling. However, you trade off the flexibility of contrastive pretraining: GenLIP requires aligned image-text pairs during pretraining (can't easily leverage unpaired data), and the visual encoder becomes somewhat specialized to language prediction rather than learning generic visual features. In production MLLMs, this means faster iteration during development and lower GPU hours for pretraining, but less ability to reuse the vision encoder for non-language downstream tasks.
Limitations and When Not to Use This
GenLIP assumes high-quality aligned image-text data is available during pretraining, which limits its applicability to datasets like Laion or proprietary datasets with good captions—it cannot leverage the large amounts of unlabeled image data that contrastive methods can exploit. The paper does not address what happens when visual-linguistic alignment is poor or when images contain content outside the language model's domain, potentially leading to degraded representations. Scalability claims are not fully substantiated in the abstract; it's unclear whether the approach maintains advantages at truly large model scales (e.g., 1B+ parameter ViTs) or only up to moderate sizes. The approach is optimized for vision-to-language tasks within MLLMs and may not generalize well to other modality pairs (video-text, audio-text) where visual grounding is less direct, leaving questions about whether this is a general principle or specific to vision-language alignment.
Research Context
GenLIP builds on the MLLM pretraining tradition (e.g., CLIP, Flamingo, LLaVA) but diverges by removing contrastive learning entirely in favor of generative modeling, echoing design principles from recent work on unified transformer architectures. It responds to a known pain point in contrastive vision-language pretraining: the requirement for large synchronized batches and careful negative sampling, which becomes a bottleneck at scale. The work aligns with a broader trend toward simplification in foundation models—similar to how GPT-2 showed that simpler next-token prediction can outcompete complex multi-task objectives, GenLIP suggests that direct language modeling may be more efficient than contrastive objectives for vision-language alignment. This opens a new research direction: whether other modality pairs or downstream tasks benefit from generative pretraining over contrastive approaches, and whether architectural unification of encoders is a general principle for multimodal models.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::