TextLDM: Language Modeling with Continuous Latent Diffusion
:::info Stub — Full Engineering Breakdown Coming This paper was featured on Hugging Face Daily Papers on 2026-05-08 with 9 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Jiaxiu Jiang et al. |
| Year | 2026 |
| HF Upvotes | 9 |
| arXiv | 2605.07748 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Diffusion Transformers (DiT) trained with flow matching in a VAE latent space have unified visual generation across images and videos. A natural next step toward a single architecture for both generation (visual synthesis) and understanding (text generation) is to apply this framework to language modeling. We propose TextLDM, which transfers the visual latent diffusion recipe to text generation with minimal architectural modification. A Transformer-based VAE maps discrete tokens to continuous latents, enhanced by Representation Alignment (REPA) with a frozen pretrained language model to produce representations effective for conditional denoising. A standard DiT then performs flow matching in this latent space, identical in architecture to its visual counterpart. The central challenge we address is obtaining high-quality continuous text representations: we find that reconstruction fidelity alone is insufficient, and that aligning latent features with a pretrained language model via REPA is critical for downstream generation quality. Trained from scratch on OpenWebText2, TextLDM substantially outperforms prior diffusion language models and matches GPT-2 under the same settings. Our results establish that the visual DiT recipe transfers effectively to language, taking a concrete step toward unified diffusion architectures for multimodal generation and understanding.
Engineering Breakdown
Plain English
TextLDM applies the diffusion transformer approach that works well for image and video generation to text generation by mapping discrete tokens into a continuous latent space using a VAE, then performing flow matching in that space. The key innovation is using Representation Alignment (REPA) with a frozen pretrained language model to ensure the learned latent representations are effective for denoising, allowing them to reuse the same DiT architecture from vision models without modification.
Key Engineering Insight
The critical insight is that you can bridge discrete token-based language modeling and continuous diffusion by aligning VAE latent representations with a frozen pretrained LM during training—this lets you use identical architecture and training procedures across vision and language domains without task-specific engineering.
Why It Matters for Engineers
For production teams, this matters because it potentially eliminates the need to maintain separate generation pipelines for different modalities—you could theoretically train and serve a single DiT-based architecture for both vision and language. It also suggests that diffusion-based generation might be competitive with autoregressive language models, which would change how you'd architect a generative AI service.
Research Context
Prior work unified visual generation across images and videos using diffusion transformers in latent space, but language modeling remained stuck in the autoregressive paradigm. This paper closes that gap by showing the latent diffusion recipe transfers to text, advancing toward a true multimodal unified architecture and suggesting diffusion could be a viable alternative to transformer-based autoregressive generation for language tasks.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::