Representation Forcing for Bottleneck-Free Unified Multimodal Models
:::info Stub — Full Engineering Breakdown Coming This paper was auto-fetched from arXiv on 2026-06-01. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Yuqing Wang et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2605.31604 |
| Download | |
| Categories | cs.CV |
Abstract
Unified multimodal models (UMMs) aim to handle perception and generation in a single model. Yet existing UMMs still rely on a frozen, separately pretrained VAE for image generation, imposing a structural bottleneck. Naively removing it introduces a quality gap, as the model must learn both high-level structure and low-level details from raw pixels. In this paper, we propose Representation Forcing (RF), a technique that closes this gap by making representation prediction a native capability of the model. Concretely, RF forces the decoder to autoregressively predict visual representations as intermediate tokens before pixels; these tokens then stay in context to guide pixel diffusion within the same backbone. By turning representations from perception outputs into generation targets, RF eliminates the need for any external generative latent space. We find that RF benefits both understanding and generation. On image generation, our pixel-space model with RF matches state-of-the-art VAE-based unified models. On image understanding, pixel-space RF generally outperforms its VAE-based variant. Together, these results offer an effective step toward end-to-end, bottleneck-free UMMs.
Engineering Breakdown
The Problem
Naively removing it introduces a quality gap, as the model must learn both high-level structure and low-level details from raw pixels. By turning representations from perception outputs into generation targets, RF eliminates the need for any external generative latent space.
The Approach
In this paper, we propose Representation Forcing (RF), a technique that closes this gap by making representation prediction a native capability of the model.
Key Results
On image generation, our pixel-space model with RF matches state-of-the-art VAE-based unified models.
Research Areas
This paper contributes to the following areas of AI/ML engineering:
- Image recognition
- Object detection
- Visual transformers
- Convolutional networks
- Multimodal learning
- Representation
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::