Coevolving Representations in Joint Image-Feature Diffusion
| Authors | Theodoros Kouzelis et al. |
| Year | 2026 |
| HF Upvotes | 3 |
| arXiv | 2604.17492 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Joint image-feature generative modeling has recently emerged as an effective strategy for improving diffusion training by coupling low-level VAE latents with high-level semantic features extracted from pre-trained visual encoders. However, existing approaches rely on a fixed representation space, constructed independently of the generative objective and kept unchanged during training. We argue that the representation space guiding diffusion should itself adapt to the generative task. To this end, we propose Coevolving Representation Diffusion (CoReDi), a framework in which the semantic representation space evolves during training by learning a lightweight linear projection jointly with the diffusion model. While naively optimizing this projection leads to degenerate solutions, we show that stable coevolution can be achieved through a combination of stop-gradient targets, normalization, and targeted regularization that prevents feature collapse. This formulation enables the semantic space to progressively specialize to the needs of image synthesis, improving its complementarity with image latents. We apply CoReDi to both VAE latent diffusion and pixel-space diffusion, demonstrating that adaptive semantic representations improve generative modeling across both settings. Experiments show that CoReDi achieves faster convergence and higher sample quality compared to joint diffusion models operating in fixed representation spaces.
Engineering Breakdown
Plain English
This paper addresses a fundamental limitation in joint image-feature diffusion models: existing approaches use a fixed representation space that doesn't adapt to the generative task. The authors propose Coevolving Representation Diffusion (CoReDi), which learns a lightweight linear projection alongside the diffusion model, allowing the semantic representation space to evolve during training. The key insight is that coupling high-level semantic features from pre-trained encoders with low-level VAE latents works better when the projection between them is task-optimized rather than frozen. The paper identifies that naive joint optimization leads to degenerate solutions and proposes mechanisms to prevent this pathology.
Core Technical Contribution
CoReDi's core novelty is treating the representation space as a learnable component rather than a static artifact. Instead of extracting semantic features once using a pre-trained encoder and keeping them fixed, the framework jointly optimizes a lightweight linear projection that transforms these features while training the diffusion model. This is fundamentally different from prior work that decouples representation learning from generative modeling—here they are tightly coupled and coevolve. The paper's main technical contribution is identifying and solving the degeneracy problem that arises when naively optimizing this projection, likely through regularization or architectural constraints that preserve representation quality.
How It Works
CoReDi operates on multi-modal input: low-level VAE latents (the compressed image representation) and high-level semantic features extracted by a pre-trained visual encoder like CLIP or a similar architecture. During diffusion training, instead of applying a fixed, pre-computed projection to these semantic features, the model learns a lightweight linear transformation matrix that maps the frozen encoder outputs to a task-optimized representation space. This projection layer is jointly optimized with the diffusion model's score network through the standard diffusion training objective (predicting noise at different timesteps). The key mechanism is that as the diffusion model learns to generate better images, it simultaneously provides gradient signals that reshape the projection, causing the semantic representation space to adapt—the 'coevolution.' To prevent collapse into degenerate solutions (where the projection becomes singular or information is lost), the framework likely enforces constraints such as orthogonality preservation, mutual information bounds, or auxiliary reconstruction losses that maintain the expressiveness of the transformed features.
Production Impact
For production image generation systems, CoReDi offers a concrete efficiency and quality improvement: you can use cheaper, smaller pre-trained encoders (or even frozen ones) and let the projection layer adapt rather than retraining large encoders end-to-end. This reduces computational cost during training since you're only learning a small linear transformation (potentially thousands of parameters versus millions in a full encoder). The approach would integrate into existing diffusion pipelines by replacing the fixed feature extraction step with a learnable projection layer, requiring minimal architectural changes. However, the degeneracy problem suggests that naïve implementations may require careful regularization tuning—engineers would need to validate that their specific encoder/diffusion combination doesn't collapse during training, and ablations show what constraints are essential. This is valuable for practitioners building multi-modal generation systems where pretrained encoders are already part of the pipeline but representation coupling is suboptimal.
Limitations and When Not to Use This
The paper's abstract doesn't reveal what mechanisms prevent degeneracy, so the practical stability of the approach remains unclear—it may require domain-specific tuning of regularization hyperparameters that don't generalize across different encoders or diffusion architectures. The framework assumes access to high-quality pre-trained visual encoders; performance on domains without good off-the-shelf models is unexplored. It's unclear how well this scales to very large diffusion models or whether the linear projection assumption becomes restrictive for complex semantic transformations. The paper doesn't address what happens if the pre-trained encoder itself is misaligned with the generation task, or how to diagnose representation degeneracy in practice—practitioners may struggle to implement this without extensive validation infrastructure.
Research Context
This work builds on the recent trend of joint image-feature generative modeling, which recognized that coupling VAE latents with semantic features from models like CLIP improves training stability and sample quality compared to diffusion on raw pixels or latents alone. It extends prior research on adaptive representations in neural networks by applying the principle specifically to diffusion models. The paper likely benchmarks against fixed-projection baselines on standard generation tasks (ImageNet, MS-COCO) to demonstrate quality improvements. This opens a research direction toward dynamic representation learning in generative models more broadly—the insight that generation objectives should inform representation space design could apply to other modalities (text, audio) and other generative paradigms (flow matching, discrete diffusion).
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::