Think in Strokes, Not Pixels: Process-Driven Image Generation via Interleaved Reasoning
| Authors | Lei Zhang et al. |
| Year | 2026 |
| HF Upvotes | 66 |
| arXiv | 2604.04746 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Humans paint images incrementally: they plan a global layout, sketch a coarse draft, inspect, and refine details, and most importantly, each step is grounded in the evolving visual states. However, can unified multimodal models trained on text-image interleaved datasets also imagine the chain of intermediate states? In this paper, we introduce process-driven image generation, a multi-step paradigm that decomposes synthesis into an interleaved reasoning trajectory of thoughts and actions. Rather than generating images in a single step, our approach unfolds across multiple iterations, each consisting of 4 stages: textual planning, visual drafting, textual reflection, and visual refinement. The textual reasoning explicitly conditions how the visual state should evolve, while the generated visual intermediate in turn constrains and grounds the next round of textual reasoning. A core challenge of process-driven generation stems from the ambiguity of intermediate states: how can models evaluate each partially-complete image? We address this through dense, step-wise supervision that maintains two complementary constraints: for the visual intermediate states, we enforce the spatial and semantic consistency; for the textual intermediate states, we preserve the prior visual knowledge while enabling the model to identify and correct prompt-violating elements. This makes the generation process explicit, interpretable, and directly supervisable. To validate proposed method, we conduct experiments under various text-to-image generation benchmarks.
Engineering Breakdown
Plain English
This paper introduces process-driven image generation, a multi-step approach that mimics how humans create paintings through iterative refinement rather than single-shot generation. Instead of generating a complete image in one step, the method interleaves textual reasoning (planning and reflection) with visual actions (drafting and refinement) across multiple iterations. The core insight is that unified multimodal models trained on text-image datasets can learn to imagine intermediate visual states, with each step's textual reasoning explicitly conditioning how the visual output should evolve. This decomposition into four stages—textual planning, visual drafting, textual reflection, and visual refinement—creates a chain-of-thought-like trajectory for image synthesis.
Core Technical Contribution
The paper's main novelty is formalizing image generation as a multi-step process with explicit interleaving of language and vision modalities, rather than treating it as a single forward pass. The key technical contribution is the four-stage iterative loop where textual reasoning acts as a conditioning signal for visual refinement, enabling the model to plan globally and refine locally in a human-like manner. This represents a shift from end-to-end single-step generation to a process-driven paradigm where intermediate reasoning states are explicitly generated and evaluated. The architecture likely leverages existing multimodal foundation models (similar to GPT-4V scale systems) but orchestrates them through a structured reasoning trajectory that was not previously explored in image generation.
How It Works
The system operates in multiple iterations, each containing four sequential stages. First, textual planning generates a description of the desired visual state and next steps. Second, visual drafting uses this plan to generate or modify the image representation. Third, textual reflection examines the current visual state and identifies gaps or needed refinements. Fourth, visual refinement applies corrections based on the reflection. The input is an initial text prompt and optionally a seed image; the output progresses through multiple intermediate visual states toward the final image. Each stage's output feeds into the next stage's input, creating a dependency chain where language explicitly guides vision transformations. The multimodal model alternates between text generation (LLM-like) and image generation/editing capabilities, likely using a unified transformer architecture that can process and output both modalities.
Production Impact
For production systems, this approach trades single-pass latency for output quality and controllability—generating an image now requires multiple forward passes (4 stages × N iterations) instead of one, potentially 4-10x increase in inference time depending on iteration depth. The benefit is significantly better alignment with user intent and the ability to provide intermediate outputs for user inspection and feedback in interactive applications. Integration would require managing a more complex generation pipeline with state management across iterations, and the ability to pause/resume or accept/reject intermediate drafts. For applications like collaborative design tools, content creation platforms, or professional asset generation, the iterative nature with explicit reasoning traces provides valuable explainability and human-in-the-loop opportunities that single-step models cannot offer. However, the compute cost per image increases substantially, requiring careful consideration of which use cases justify the overhead.
Limitations and When Not to Use This
The paper's approach assumes that multimodal models have sufficient capability to perform both high-quality visual reasoning and visual generation, which may not hold for smaller or less capable base models. The iterative nature introduces cumulative error risk where mistakes in early stages (planning or drafting) can compound through refinement stages, potentially requiring careful loss function design or rejection sampling to prevent degradation. The paper doesn't address how to determine the optimal number of iterations—too few may leave images unrefined, too many wastes compute without quality gain. The method likely requires substantially more training data or fine-tuning to learn the interleaving behavior, and it's unclear if standard text-image datasets provide sufficient signal for the intermediate reasoning stages to be useful. Real-world performance on diverse prompts, edge cases, and the actual wall-clock latency improvements in practice remain empirically unvalidated in the abstract.
Research Context
This work builds on the recent success of chain-of-thought reasoning in language models (Wei et al., 2022) and extends the concept to multimodal generation, treating image synthesis as a reasoning problem rather than pure pattern matching. It relates to prior work on iterative image editing and refinement (e.g., Prompt-to-Prompt, Pix2Pix Cycle consistency) but differs by making the reasoning process explicit and learned rather than hand-engineered. The paper contributes to the emerging field of process-aware or trajectory-based generation, which has shown promise in planning and reinforcement learning domains. It opens research directions into learning what makes good intermediate states, how to evaluate reasoning quality in vision tasks, and whether chain-of-thought-like decomposition generalizes beyond language to multimodal tasks.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::