HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds
| Authors | Team HY-World et al. |
| Year | 2026 |
| HF Upvotes | 93 |
| arXiv | 2604.14268 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
We introduce HY-World 2.0, a multi-modal world model framework that advances our prior project HY-World 1.0. HY-World 2.0 accommodates diverse input modalities, including text prompts, single-view images, multi-view images, and videos, and produces 3D world representations. With text or single-view image inputs, the model performs world generation, synthesizing high-fidelity, navigable 3D Gaussian Splatting (3DGS) scenes. This is achieved through a four-stage method: a) Panorama Generation with HY-Pano 2.0, b) Trajectory Planning with WorldNav, c) World Expansion with WorldStereo 2.0, and d) World Composition with WorldMirror 2.0. Specifically, we introduce key innovations to enhance panorama fidelity, enable 3D scene understanding and planning, and upgrade WorldStereo, our keyframe-based view generation model with consistent memory. We also upgrade WorldMirror, a feed-forward model for universal 3D prediction, by refining model architecture and learning strategy, enabling world reconstruction from multi-view images or videos. Also, we introduce WorldLens, a high-performance 3DGS rendering platform featuring a flexible engine-agnostic architecture, automatic IBL lighting, efficient collision detection, and training-rendering co-design, enabling interactive exploration of 3D worlds with character support. Extensive experiments demonstrate that HY-World 2.0 achieves state-of-the-art performance on several benchmarks among open-source approaches, delivering results comparable to the closed-source model Marble. We release all model weights, code, and technical details to facilitate reproducibility and support further research on 3D world models.
Engineering Breakdown
Plain English
HY-World 2.0 is a multi-modal world model that generates navigable 3D scenes from text prompts or images by combining four specialized stages: panorama generation, trajectory planning, stereo reconstruction, and scene composition. The system accepts diverse inputs—text, single images, multi-view images, and videos—and outputs high-fidelity 3D Gaussian Splatting (3DGS) representations that can be explored interactively. This represents a significant upgrade from HY-World 1.0, with improved panorama quality, better 3D scene understanding, and enhanced stereo depth estimation. The approach solves the practical problem of converting sparse 2D input into dense, traversable 3D environments without requiring extensive 3D training data.
Core Technical Contribution
The core innovation is a four-stage pipeline architecture that decomposes world generation into specialized, interpretable modules rather than relying on end-to-end learning. HY-Pano 2.0 improves panorama synthesis fidelity through enhanced generation techniques; WorldNav introduces explicit trajectory planning to guide 3D expansion in semantically meaningful directions; WorldStereo 2.0 refines keyframe-based stereo reconstruction for better depth estimation; and WorldMirror 2.0 composes these outputs into coherent 3D Gaussian splat representations. This modular design allows each component to be independently improved and provides interpretability about what regions are generated versus inferred, a departure from monolithic end-to-end world models that treat generation as a black box.
How It Works
The pipeline begins with the user providing either a text prompt or single image. HY-Pano 2.0 generates a 360-degree panoramic view from this sparse input, establishing the immediate visual context. WorldNav then plans a trajectory through the 3D space—determining where to place virtual camera positions to explore and expand the scene coherently rather than randomly sampling viewpoints. For each planned viewpoint, WorldStereo 2.0 synthesizes stereo image pairs and performs depth estimation to infer 3D geometry at those locations. Finally, WorldMirror 2.0 aggregates all the generated views and depth maps into a unified 3D Gaussian Splatting representation, which allows smooth navigation and rendering from arbitrary camera positions. This sequence converts a single 2D observation into a navigable 3D world by systematically filling in unobserved regions.
Production Impact
For teams building interactive 3D content generation systems, this approach offers a faster path to 3D asset creation—reducing the need for manual 3D modeling or expensive multi-view capture. The modular architecture means you can replace individual stages (e.g., swap in a better depth estimator into WorldStereo) without retraining the entire system, lowering iteration costs. However, compute cost is non-trivial: generating a single world likely requires multiple forward passes through diffusion models (for panorama and view synthesis) plus depth inference, potentially adding seconds to minutes of latency. Data requirements are also substantial—the system must be trained on large-scale 3D scene datasets and text-3D pairs. Integration challenges include managing the complexity of the four-stage pipeline, handling failure propagation (errors in trajectory planning corrupt downstream geometry), and ensuring consistent geometry across synthesized views to avoid visual artifacts during navigation.
Limitations and When Not to Use This
The system assumes that a single 360-degree panorama can adequately represent the immediate context, which breaks down for scenes with complex occlusions or multi-level structures where different camera heights reveal entirely different content. The trajectory planning stage (WorldNav) relies on heuristics or learned policies that may not generalize to unusual layouts—a narrow corridor or dense forest could generate incoherent expansion plans. The paper does not address how well the 3DGS representation handles dynamic content, reflective surfaces, or transparent materials, which are common failure modes in view synthesis. Additionally, there is no clear discussion of how the system validates geometric consistency across synthesized views or how it handles hallucinations when filling occluded regions—particularly risky in applications like architectural visualization or robotics planning where false 3D structure could lead to safety issues.
Research Context
This work builds directly on the HY-World 1.0 foundation, iterating on the core idea of staged world generation but with stronger components at each stage. It sits at the intersection of generative modeling (diffusion-based image synthesis), 3D reconstruction (Gaussian splatting, stereo depth estimation), and planning (trajectory generation for view synthesis). The research advances the broader goal of few-shot 3D scene understanding—generating complete 3D environments from minimal input—which is relevant for game development, robotics simulation, and virtual reality content creation. The emphasis on modular stages over end-to-end learning represents a philosophical shift in the field toward interpretability and composability, suggesting future work may focus on making each stage learnable or replaceable independently.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::