FlowInOne:Unifying Multimodal Generation as Image-in, Image-out Flow Matching
| Authors | Junchao Yi et al. |
| Year | 2026 |
| HF Upvotes | 10 |
| arXiv | 2604.06757 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Multimodal generation has long been dominated by text-driven pipelines where language dictates vision but cannot reason or create within it. We challenge this paradigm by asking whether all modalities, including textual descriptions, spatial layouts, and editing instructions, can be unified into a single visual representation. We present FlowInOne, a framework that reformulates multimodal generation as a purely visual flow, converting all inputs into visual prompts and enabling a clean image-in, image-out pipeline governed by a single flow matching model. This vision-centric formulation naturally eliminates cross-modal alignment bottlenecks, noise scheduling, and task-specific architectural branches, unifying text-to-image generation, layout-guided editing, and visual instruction following under one coherent paradigm. To support this, we introduce VisPrompt-5M, a large-scale dataset of 5 million visual prompt pairs spanning diverse tasks including physics-aware force dynamics and trajectory prediction, alongside VP-Bench, a rigorously curated benchmark assessing instruction faithfulness, spatial precision, visual realism, and content consistency. Extensive experiments demonstrate that FlowInOne achieves state-of-the-art performance across all unified generation tasks, surpassing both open-source models and competitive commercial systems, establishing a new foundation for fully vision-centric generative modeling where perception and creation coexist within a single continuous visual space.
Engineering Breakdown
Plain English
FlowInOne presents a fundamentally different approach to multimodal generation by treating all inputs—text descriptions, spatial layouts, editing instructions—as visual representations rather than separate modalities. Instead of the traditional text-to-image pipeline where language dominates, this framework converts everything into visual prompts and routes them through a single flow matching model in an image-in, image-out architecture. The key insight is that unifying all modalities into a visual domain eliminates cross-modal alignment bottlenecks, complex noise scheduling, and task-specific branches that plague current systems. This vision-centric formulation enables a clean, general-purpose pipeline that handles text-to-image generation, layout-guided editing, and visual instruction following with a single model.
Core Technical Contribution
The core novelty is reformulating multimodal generation as a purely visual flow matching problem rather than a text-conditioned one. Instead of encoding text into embeddings and using them to condition a diffusion or flow model, FlowInOne converts all input modalities (text, layouts, instructions, sketches) into visual token representations first, creating a genuine image-in, image-out pipeline. This eliminates the architectural asymmetry where language is treated as a privileged conditioning input—all information flows through the same visual channel. By operating entirely in visual space, the framework naturally sidesteps cross-modal alignment issues, temporal noise scheduling complications, and the need for task-specific decoder heads or adapter modules that fragment current multimodal systems.
How It Works
The pipeline begins by converting diverse input modalities into a unified visual representation space: text is encoded into visual tokens (likely using a learned embedding or tokenizer), spatial layouts are rendered as visual tokens encoding structure, and editing instructions are similarly visualized. These visual prompt tokens are concatenated or merged into a single visual context representation. A flow matching model then operates on this purely visual input, learning to match the probability flow from noise to clean image space conditioned only on this visual context. During generation, the model iteratively refines a noisy image by following the learned flow trajectory, guided by the visual prompt tokens derived from all input modalities. The output is a clean image that satisfies the combined constraints from all input modalities, with no need for separate text encoders, cross-attention mechanisms, or modality-specific branching in the architecture.
Production Impact
Adopting FlowInOne would dramatically simplify multimodal generation pipelines in production systems. Instead of maintaining separate text-to-image, layout-to-image, and instruction-following models (or complex conditional branches within one model), teams deploy a single flow matching model that handles all tasks through unified visual prompting. This reduces model complexity, inference latency (single forward pass through one model versus orchestrating multiple encoders), and memory footprint, making it more practical for resource-constrained environments like mobile or edge devices. The elimination of cross-modal alignment bottlenecks also improves generation quality and consistency across different input types—text descriptions and layouts won't conflict or suffer from misalignment. However, the approach requires solving the non-trivial problem of converting all modalities (especially text and abstract instructions) into meaningful visual representations, which may require task-specific tokenizers or encoders that add hidden complexity, and the assumption that all modalities benefit from visual representation may not hold for truly abstract or semantic reasoning tasks.
Limitations and When Not to Use This
The paper's vision-centric approach assumes that all meaningful information in multimodal generation can be effectively represented visually, which may not hold for abstract reasoning, complex semantic instructions, or modalities that don't naturally map to spatial/visual tokens. Converting text descriptions into visual tokens is a lossy process that may discard important linguistic nuances or compositional semantics that current text-to-image models exploit through language embeddings. The framework doesn't address whether this approach scales to very long or complex instructions (how many visual tokens are needed to encode an intricate architectural blueprint or a paragraph-long narrative?), and there's no discussion of how to handle modalities with truly different semantic properties (e.g., audio, temporal sequences, or highly abstract categorical metadata). Additionally, the paper lacks evaluation on out-of-distribution or adversarial inputs, so it's unclear how robust the visual tokenization approach is when given unusual or conflicting input modalities.
Research Context
FlowInOne builds on the recent shift from diffusion models to flow matching in generative modeling (Liphardt et al., Chen et al.), which offers cleaner training dynamics and more efficient sampling. It directly challenges the dominant paradigm in multimodal generation (established by CLIP-based text-to-image models like DALL-E, Stable Diffusion, and Imagen) where text embeddings condition the visual generation process. The work is motivated by observations that text-conditioned architectures create alignment problems between linguistic and visual semantics, and that multi-task models require architectural compromises. This research opens a new direction: exploring whether other modalities (and even text itself) are better served as visual prompts than as separate conditioning channels, potentially leading to future work on optimal visual tokenization schemes, cross-modal translation in unified visual spaces, and more efficient architectures for handling diverse input types without branching or gating mechanisms.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::