LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling
| Authors | Yuxin Chen et al. |
| Year | 2026 |
| HF Upvotes | 14 |
| arXiv | 2604.11748 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Continuous diffusion has been the foundation of high-fidelity, controllable, and few-step generation of many data modalities such as images. However, in language modeling, prior continuous diffusion language models (DLMs) lag behind discrete counterparts due to the sparse data space and the underexplored design space. In this work, we close this gap with LangFlow, the first continuous DLM to rival discrete diffusion, by connecting embedding-space DLMs to Flow Matching via Bregman divergence, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, as we find it improves both likelihood and sample quality of embedding-space DLMs with effects substantially different from discrete diffusion. Putting everything together, LangFlow rivals top discrete DLMs on both the perplexity (PPL) and the generative perplexity (Gen. PPL), reaching a PPL of 30.0 on LM1B and 24.6 on OpenWebText. It even exceeds autoregressive baselines in zero-shot transfer on 4 out of 7 benchmarks. LangFlow provides the first clear evidence that continuous diffusion is a promising paradigm for language modeling. Homepage: https://github.com/nealchen2003/LangFlow
Engineering Breakdown
Plain English
LangFlow introduces the first continuous diffusion language model that matches the performance of discrete diffusion models, closing a gap that has existed because continuous diffusion works well for images but poorly for text. The paper connects embedding-space diffusion models to Flow Matching using Bregman divergence and proposes three key innovations: an ODE-based bound for properly evaluating continuous flow models, an information-uniform noise schedule principle, and a learnable Gumbel-based noise scheduler. This is significant because continuous diffusion could offer advantages in controllability and few-step generation for language tasks if the performance gap could be closed, which LangFlow appears to do.
Core Technical Contribution
The core novelty is bridging the theoretical and practical gap between continuous flow-based models and discrete diffusion for language by reformulating the problem through Bregman divergence, which allows proper handling of discrete token spaces in continuous latent embeddings. The paper derives a novel ODE-based negative log-likelihood (NLL) bound that provides principled evaluation criteria specifically for continuous flow-based language models, replacing ad-hoc evaluation methods. The information-uniform principle for noise scheduling is a theoretical contribution that ensures consistent information content across diffusion steps, leading to the design of a learnable noise scheduler parameterized by a Gumbel distribution instead of fixed schedules. These three components work together to address why prior continuous DLMs underperformed: poor theoretical grounding, misaligned noise schedules for discrete data, and lack of adaptability.
How It Works
LangFlow operates by embedding discrete tokens into a continuous space, where the diffusion process becomes mathematically tractable through flow matching. The model takes input tokens, encodes them into embeddings, and applies a noise scheduler that gradually adds noise according to the information-uniform principle—ensuring that at each timestep, the mutual information between the noisy embedding and the original is balanced rather than following standard Gaussian schedules. The core mechanism uses Bregman divergence to measure distances in the embedding space, which is more appropriate for non-Euclidean geometry induced by embedding metrics. A learnable noise scheduler parameterized by Gumbel distributions adapts the noise level dynamically during training, replacing fixed schedules like linear or cosine decay. The model predicts clean embeddings from noisy ones using a neural network trained with the ODE-based NLL bound, which provides a tighter and more principled loss function than standard diffusion objectives. At inference, the reverse process denoises embeddings step-by-step before decoding back to discrete tokens.
Production Impact
In production, this approach enables language models with explicit controllability over generation—users can steer outputs by conditioning diffusion at specific noise levels, similar to image diffusion models' strength. The ODE-based NLL bound provides a more reliable metric for model evaluation and comparison, reducing the need for expensive empirical benchmarking and giving better confidence in model quality before deployment. Training becomes more efficient because the learnable noise scheduler adapts to the data distribution, potentially reducing the number of diffusion steps or training iterations needed to converge compared to fixed schedules. However, adoption requires replacing your discrete tokenization and generation pipeline with continuous embedding-space operations, which may increase inference latency (each denoising step requires a forward pass) unless you optimize for parallel decoding or use distillation. Integration complexity is non-trivial: you need to maintain both an embedding space and token decoder, handle the ODE solver during inference, and validate that the Bregman divergence assumption holds for your vocabulary size and embedding dimension.
Limitations and When Not to Use This
The paper assumes that discrete tokens can be meaningfully embedded in a low-dimensional continuous space where Bregman divergence is appropriate, which may break down for languages with very large vocabularies or highly structured token relationships. Inference cost is likely higher than autoregressive models because each generation requires multiple denoising steps with full forward passes, and the paper doesn't provide concrete latency comparisons against GPT-style models or discuss step count trade-offs. The learnable Gumbel-based noise scheduler adds hyperparameters and may require careful tuning per task or domain—the paper doesn't fully explore sensitivity to these choices or provide guidance on when the approach fails. The work focuses on language modeling perplexity/NLL but doesn't thoroughly evaluate downstream task performance (classification, QA, generation quality metrics), so it's unclear if the theoretical improvements translate to practical benefits for deployed systems.
Research Context
This paper builds on a decade of diffusion model research that succeeded spectacularly in vision (DDPM, Stable Diffusion) but struggled in NLP, where discrete token spaces and high-dimensional output spaces posed challenges that autoregressive models solved more elegantly. It extends recent work on Flow Matching (Liphardt et al., Albergo & Vanden-Eijnden) to language and bridges embedding-space diffusion models to information-theoretic principles, advancing the theoretical understanding of why discrete diffusion underperformed. The work opens a research direction toward hybrid generation paradigms that combine diffusion's controllability with language model scale and performance, potentially enabling applications like controllable text generation, few-shot adaptation, and resolution-adaptive decoding. This is particularly timely as multimodal and control-centric language tasks become more important than pure perplexity optimization.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::