World2Minecraft: Occupancy-Driven Simulated Scenes Construction
| Authors | Lechao Zhang et al. |
| Year | 2026 |
| HF Upvotes | 3 |
| arXiv | 2604.27578 |
| Download | |
| Code | https://github.com/Nepenthes-zlc/World2Minecraft |
Abstract
Embodied intelligence requires high-fidelity simulation environments to support perception and decision-making, yet existing platforms often suffer from data contamination and limited flexibility. To mitigate this, we propose World2Minecraft to convert real-world scenes into structured Minecraft environments based on 3D semantic occupancy prediction. In the reconstructed scenes, we can effortlessly perform downstream tasks such as Vision-Language Navigation(VLN). However, we observe that reconstruction quality heavily depends on accurate occupancy prediction, which remains limited by data scarcity and poor generalization in existing models. We introduce a low-cost, automated, and scalable data acquisition pipeline for creating customized occupancy datasets, and demonstrate its effectiveness through MinecraftOcc, a large-scale dataset featuring 100,165 images from 156 richly detailed indoor scenes. Extensive experiments show that our dataset provides a critical complement to existing datasets and poses a significant challenge to current SOTA methods. These findings contribute to improving occupancy prediction and highlight the value of World2Minecraft in providing a customizable and editable platform for personalized embodied AI research. Project page:https://world2minecraft.github.io/.
Engineering Breakdown
Plain English
This paper presents World2Minecraft, a system that converts real-world 3D scenes into reconstructed Minecraft environments using 3D semantic occupancy prediction. The core problem is that embodied AI systems need high-fidelity simulation environments for perception and navigation tasks, but existing platforms suffer from data contamination and lack flexibility. The authors propose an automated pipeline that predicts occupancy from real-world data and reconstructs scenes in Minecraft, enabling downstream tasks like Vision-Language Navigation. They also introduce a scalable data acquisition pipeline to address the data scarcity problem that limits occupancy prediction model generalization.
Core Technical Contribution
The primary technical novelty is the occupancy-driven scene reconstruction pipeline that bridges real-world 3D data to structured Minecraft environments. Rather than attempting direct voxel-to-block conversion, the authors use 3D semantic occupancy prediction as an intermediate representation that preserves spatial structure while enabling clean simulation. A secondary contribution is the low-cost, automated data acquisition pipeline for generating customized occupancy datasets without manual annotation overhead. This addresses a fundamental bottleneck: occupancy prediction models fail in new environments due to distribution shift, and the paper's data pipeline provides a scalable solution to create domain-specific training data at minimal cost.
How It Works
The system operates in three stages: First, it acquires real-world 3D scene data (likely from RGB-D sensors, LiDAR, or photogrammetry) and creates a point cloud or mesh representation. Second, it runs a 3D semantic occupancy prediction model on this data, which classifies each voxel in the scene space into semantic categories (free space, wall, object type, etc.) and predicts occupancy probability. Third, it converts the occupancy grid into Minecraft blocks by mapping semantic categories to appropriate block types while respecting Minecraft's discrete voxel grid constraints. The data acquisition pipeline operates independently: it automatically captures diverse scene configurations, generates ground-truth occupancy annotations (possibly through multi-view consistency or LiDAR fusion), and creates dataset splits. The reconstructed Minecraft environments can then be used for training embodied AI agents on tasks like Vision-Language Navigation, where the structured simulation environment provides deterministic physics and clean visual rendering.
Production Impact
For teams building embodied AI systems, this approach solves a critical infrastructure problem: obtaining diverse, high-quality simulation environments without expensive manual design. Instead of hiring designers to craft Minecraft worlds or struggling with simulator-to-reality transfer, engineers can now automatically generate training environments from real-world captures. This dramatically reduces iteration time when deploying navigation agents to new buildings or outdoor scenes—capture once, reconstruct automatically, train immediately. The trade-offs are significant: occupancy prediction quality directly limits reconstruction fidelity, requiring investment in robust prediction models; the pipeline requires real-world capture infrastructure (cameras, depth sensors); and Minecraft's discrete grid may lose fine geometric details from continuous real-world data. Compute-wise, occupancy prediction is non-trivial (3D convolutions or sparse networks across large volumes), though this is a one-time cost per scene.
Limitations and When Not to Use This
The paper's core limitation is its dependence on occupancy prediction quality—the pipeline cannot recover from poor predictions, meaning generalization failures propagate downstream. The Minecraft discrete voxel representation inherently loses geometric detail from continuous real-world data, which may degrade visual realism and limit applicability to tasks requiring fine spatial reasoning. The data acquisition pipeline's generalization to entirely new scene types (e.g., outdoor wilderness scenes after training on indoor offices) is unproven; the paper mentions data scarcity as a limitation but doesn't fully solve it—their pipeline generates data but still requires initial real-world captures. Additionally, the approach assumes scenes can be reasonably represented in Minecraft's limited block palette and scale constraints; complex scenes with fine geometry, transparent materials, or dynamic elements may resist clean reconstruction.
Research Context
This work builds on the embodied AI renaissance where researchers realized simulation environments (Habitat, Minecraft, AirSim) are critical for training agents at scale before deploying to robots or real homes. It advances the Vision-Language Navigation task, where agents must follow natural language instructions while perceiving their environment—a benchmark where photorealistic simulators have driven recent progress. The paper extends occupancy prediction, a technique popularized in 3D scene understanding and robotics (e.g., for robot navigation planning), into a bridge between real and simulated domains. By coupling automatic scene capture with occupancy prediction and Minecraft reconstruction, it opens a new research direction: can we build one unified pipeline that converts arbitrary real-world environments into diverse simulation scenarios for multi-agent training?
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::