CLEAR: Unlocking Generative Potential for Degraded Image Understanding in Unified Multimodal Models
| Authors | Xiangzhao Hao et al. |
| Year | 2026 |
| HF Upvotes | 10 |
| arXiv | 2604.04780 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Image degradation from blur, noise, compression, and poor illumination severely undermines multimodal understanding in real-world settings. Unified multimodal models that combine understanding and generation within a single architecture are a natural fit for this challenge, as their generative pathway can model the fine-grained visual structure that degradation destroys. Yet these models fail to leverage their own generative capacity on degraded inputs. We trace this disconnect to two compounding factors: existing training regimes never ask the model to invoke generation during reasoning, and the standard decode-reencode pathway does not support effective joint optimization. We present CLEAR, a framework that connects the two capabilities through three progressive steps: (1) supervised fine-tuning on a degradation-aware dataset to establish the generate-then-answer reasoning pattern; (2) a Latent Representation Bridge that replaces the decode-reencode detour with a direct, optimizable connection between generation and reasoning; (3) Interleaved GRPO, a reinforcement learning method that jointly optimizes text reasoning and visual generation under answer-correctness rewards. We construct MMD-Bench, covering three degradation severity levels across six standard multimodal benchmarks. Experiments show that CLEAR substantially improves robustness on degraded inputs while preserving clean-image performance. Our analysis further reveals that removing pixel-level reconstruction supervision leads to intermediate visual states with higher perceptual quality, suggesting that task-driven optimization and visual quality are naturally aligned.
Engineering Breakdown
Plain English
This paper addresses a critical gap in multimodal AI systems: they struggle with degraded images (blurry, noisy, compressed, poorly lit) in real-world deployment. The authors propose CLEAR, a framework that enables unified multimodal models to leverage their built-in generative capabilities to restore image quality while simultaneously performing understanding tasks. The key insight is that existing models have generation capacity but never use it during reasoning on degraded inputs, and their training pipelines don't optimize for this joint capability. CLEAR introduces a three-step progressive training approach that bridges understanding and generation in a single unified architecture.
Core Technical Contribution
The core novelty is identifying and solving a fundamental mismatch in unified multimodal models: they possess generative pathways that can reconstruct fine-grained visual structure lost to degradation, but training regimes never condition reasoning tasks on invoked generation. Rather than treating image restoration as a preprocessing step separate from understanding, CLEAR embeds it directly into the model's inference flow through progressive supervised fine-tuning, where the model learns to call generation operations during reasoning. The technical breakthrough is reformulating the decode-reencode pathway to support joint end-to-end optimization of both restoration and downstream understanding, overcoming prior architectural constraints that prevented effective backpropagation through regenerated content.
How It Works
CLEAR operates through three progressive training phases that incrementally teach the unified multimodal model to invoke its own generative capabilities. Phase one uses supervised fine-tuning where the model explicitly learns which degraded inputs require generation and how to invoke restoration sub-modules before reasoning. Phase two (not fully described in the abstract) likely involves mixing understanding and generation tasks during training so the model learns when restoration helps downstream performance. Phase three progressively increases task complexity and degradation severity. During inference on a degraded input, the model first detects degradation type, invokes the appropriate generative pathway to reconstruct missing visual information, re-encodes the restored image through its visual encoder, then proceeds with standard reasoning tasks—all within a single forward pass. The key architectural constraint that CLEAR addresses is enabling effective gradient flow through the regenerated content during training, which standard encode-generate-encode pipelines struggled to optimize.
Production Impact
For production systems handling real-world visual data, CLEAR eliminates the need for separate preprocessing pipelines dedicated to image restoration, reducing latency and system complexity. Instead of orchestrating external denoising, deblurring, or super-resolution models before passing images to multimodal systems, engineers deploy a single unified model that handles restoration internally and on-demand. This directly improves accuracy on degraded inputs (compression artifacts from mobile cameras, surveillance footage blur, low-light scenes) without requiring clean training data. Trade-offs include increased model size and inference compute (generation adds parameters and forward passes), longer inference latency due to optional restoration pathways, and higher peak memory during training due to joint optimization. For organizations processing high volumes of user-generated or sensor imagery, this approach could reduce preprocessing infrastructure costs while improving downstream multimodal task performance on naturally degraded inputs, though you'd need to benchmark the compute overhead against your specific quality gains.
Limitations and When Not to Use This
The paper assumes image degradation is the primary bottleneck for multimodal understanding, which may not hold for all use cases—some degradation types might be better handled through robust feature learning rather than explicit restoration. CLEAR requires the underlying multimodal model to already support high-quality generative pathways; weaker base models won't benefit as much. The framework's effectiveness depends on accurate detection of when to invoke generation (calling it unnecessarily wastes compute, while missing opportunities hurts performance), and the paper doesn't clarify how this selection mechanism is trained or its false-positive/negative rates. Additionally, the approach assumes sufficient labeled data pairing degraded images with clean targets for phase-one supervised fine-tuning, which may be expensive to collect for specialized domains. Generalization to out-of-distribution degradation types (unseen blur kernels, novel compression codecs) is untested, and the paper doesn't discuss computational scaling—how inference latency degrades when generation is invoked at scale, or whether selective generation is feasible in resource-constrained environments.
Research Context
This work builds on the emerging trend of unified multimodal architectures (like GPT-4V, LLaVA, or Flamingo) that combine vision understanding and generation in a single model, recognizing that generative capacity is a natural fit for inverse problems like restoration. It relates to prior work on image restoration networks (diffusion models, GANs) but differs by integrating restoration as an internal reasoning component rather than a standalone preprocessing step. The paper implicitly contributes to the broader challenge of making foundation models robust to distribution shift and real-world degradation—an active area since most training data comes from curated, high-quality sources. CLEAR opens a research direction around progressive training strategies for multimodal models where capabilities are unlocked in stages, and validates that on-demand generative processing during reasoning can improve both efficiency and accuracy.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::