Map2World: Segment Map Conditioned Text to 3D World Generation
| Authors | Jaeyoung Chung et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2605.00781 |
| Download | |
| Categories | cs.CV |
Abstract
3D world generation is essential for applications such as immersive content creation or autonomous driving simulation. Recent advances in 3D world generation have shown promising results; however, these methods are constrained by grid layouts and suffer from inconsistencies in object scale throughout the entire world. In this work, we introduce a novel framework, Map2World, that first enables 3D world generation conditioned on user-defined segment maps of arbitrary shapes and scales, ensuring global-scale consistency and flexibility across expansive environments. To further enhance the quality, we propose a detail enhancer network that generates fine details of the world. The detail enhancer enables the addition of fine-grained details without compromising overall scene coherence by incorporating global structure information. We design the entire pipeline to leverage strong priors from asset generators, achieving robust generalization across diverse domains, even under limited training data for scene generation. Extensive experiments demonstrate that our method significantly outperforms existing approaches in user-controllability, scale consistency, and content coherence, enabling users to generate 3D worlds under more complex conditions.
Engineering Breakdown
Plain English
Map2World addresses a key limitation in 3D world generation: existing methods are constrained by rigid grid layouts and produce inconsistent object scales across large environments. The paper proposes a framework that generates 3D worlds conditioned on user-defined semantic segment maps of arbitrary shapes and sizes, maintaining global-scale consistency throughout expansive scenes. A detail enhancer network adds fine-grained details without degrading overall scene coherence. This enables flexible, large-scale 3D world generation for applications like immersive content creation and autonomous driving simulation.
Core Technical Contribution
The core innovation is a two-stage generation framework that decouples layout control from detail synthesis. Unlike prior work that generates 3D worlds from fixed grid structures, Map2World conditions generation on arbitrary semantic maps, solving the scale consistency problem through a globally-aware generation process. The detail enhancer network is a secondary component that enriches coarse geometry with fine details through a separate conditioning pathway, preventing detail generation from degrading the macro-level spatial coherence that the main generator establishes. This modular approach—separating large-scale structure from fine details—is novel in the 3D world generation space.
How It Works
The system takes a user-defined semantic segmentation map as input, where regions specify object categories at arbitrary locations and scales. The primary generator network processes this map and produces a coarse 3D world representation that respects the spatial layout and maintains consistent scales across all objects in the scene. This coarse output captures the global structure and ensures objects at different distances maintain semantically coherent proportions. The detail enhancer then operates on regions of this coarse world, adding fine-grained geometric and textural details by conditioning on local patches while respecting the macro-scale decisions already made. The two-stage design prevents the detail generation from introducing conflicting large-scale modifications that would break scene coherence. Output is a complete 3D world scene with both structural integrity and fine details.
Production Impact
For teams building immersive content creation tools or simulation engines, this removes the grid constraint that locks layouts into discrete cells, enabling organic, free-form world design. In autonomous driving simulation pipelines, you can now generate diverse, large-scale environments with consistent object proportions automatically—critical for training perception systems on varied scenarios without manual authoring. Integration requires: (1) a semantic map authoring interface, (2) inference infrastructure for the two-stage generation (coarse + detail), and (3) output conversion to standard 3D formats (USD, FBX). Trade-offs include increased memory during detail enhancement, longer inference time versus single-stage methods, and dependency on training data that covers the distribution of maps and scenes you'll encounter in production.
Limitations and When Not to Use This
The approach assumes semantic maps are well-formed and semantically consistent—malformed or contradictory map inputs may produce incoherent worlds. The paper doesn't discuss how the method scales to extremely large worlds (100+ sq km) or whether memory and compute grow prohibitively; this is critical for open-world simulation. Training likely requires paired datasets of semantic maps and 3D worlds, which is expensive to acquire at scale; generalization to novel map topologies unseen during training is unclear. The detail enhancer, while preventing global degradation, may still produce locally inconsistent details if the coarse geometry has ambiguous regions, and there's no discussed mechanism for user control over detail level or style.
Research Context
This work builds on recent advances in diffusion-based 3D generation and conditional scene synthesis, extending them from constrained grid-based layouts to flexible semantic maps. It addresses limitations in prior 3D world generation pipelines that either operated on rigid structures or struggled with scale consistency in large scenes. The research opens directions in hierarchical 3D generation (coarse-to-fine) and semantic-conditional synthesis, moving toward user-driven, constraint-based 3D content creation. This also connects to research in scene understanding and semantic segmentation, bridging 2D layout control to 3D geometric generation.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::