Habitat-GS: A High-Fidelity Navigation Simulator with Dynamic Gaussian Splatting
| Authors | Ziyuan Xia et al. |
| Year | 2026 |
| HF Upvotes | 14 |
| arXiv | 2604.12626 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Training embodied AI agents depends critically on the visual fidelity of simulation environments and the ability to model dynamic humans. Current simulators rely on mesh-based rasterization with limited visual realism, and their support for dynamic human avatars, where available, is constrained to mesh representations, hindering agent generalization to human-populated real-world scenarios. We present Habitat-GS, a navigation-centric embodied AI simulator extended from Habitat-Sim that integrates 3D Gaussian Splatting scene rendering and drivable gaussian avatars while maintaining full compatibility with the Habitat ecosystem. Our system implements a 3DGS renderer for real-time photorealistic rendering and supports scalable 3DGS asset import from diverse sources. For dynamic human modeling, we introduce a gaussian avatar module that enables each avatar to simultaneously serve as a photorealistic visual entity and an effective navigation obstacle, allowing agents to learn human-aware behaviors in realistic settings. Experiments on point-goal navigation demonstrate that agents trained on 3DGS scenes achieve stronger cross-domain generalization, with mixed-domain training being the most effective strategy. Evaluations on avatar-aware navigation further confirm that gaussian avatars enable effective human-aware navigation. Finally, performance benchmarks validate the system's scalability across varying scene complexity and avatar counts.
Engineering Breakdown
Plain English
Habitat-GS extends the Habitat-Sim embodied AI simulator by replacing mesh-based rasterization with 3D Gaussian Splatting (3DGS) for photorealistic rendering and adding support for dynamic human avatars represented as drivable gaussians. The core problem is that current simulators produce low visual fidelity and cannot realistically simulate humans in scenes, which causes trained navigation agents to fail when deployed in real-world environments with people. The authors' solution maintains full backward compatibility with the existing Habitat ecosystem while enabling agents to train on visually realistic, human-populated scenes. This work directly addresses the sim-to-real gap in embodied AI by providing both photorealistic rendering and scalable dynamic human modeling.
Core Technical Contribution
The technical novelty is the integration of 3D Gaussian Splatting as a real-time renderer within an embodied AI framework, combined with a new representation called 'drivable gaussian avatars' for modeling dynamic humans in simulation. Prior work used either mesh-based rasterization (low visual quality) or supported humans only through mesh representations (limited realism and poor generalization). The authors' key insight is that gaussians are both fast to render at real-time speeds and expressive enough to capture human motion and appearance variations. They also engineered a scalable asset import pipeline that allows 3DGS content from diverse sources to be used within Habitat, addressing the practical bottleneck of building diverse 3D environments.
How It Works
The system takes 3D scene data and converts it into a 3DGS representation—a collection of millions of small gaussian primitives with learnable centers, colors, opacities, and covariance matrices that together represent geometry and appearance. During rendering, the gaussians are splatted onto the image plane in real-time using GPU-accelerated tile-based splatting, producing photorealistic images at interactive frame rates (typically 60+ fps). For dynamic humans, the authors propose 'drivable gaussian avatars' where a base gaussian representation is controlled by skeleton-based pose parameters; as the avatar moves through the scene (driven by navigation or animation controllers), the gaussians deform and reproject to match the new pose. The agent interacts with this rendered scene via standard vision input (RGB or RGB-D), receiving navigation rewards based on reaching waypoints, exactly as in standard Habitat environments. The entire pipeline maintains compatibility with Habitat's modular design, allowing researchers to swap rendering backends without rewriting training code.
Production Impact
For teams training embodied AI agents, this system solves two critical problems: visual domain gap (agents trained in low-fidelity meshes fail on real cameras) and human-agent interaction (agents never see humans during training, so they don't learn avoidance or social navigation). Adopting Habitat-GS would allow researchers to generate large synthetic datasets with photorealistic visuals and diverse human behaviors, dramatically reducing reliance on expensive real-world data collection. The trade-off is computational cost—3DGS rendering is faster than ray-tracing but still requires GPU acceleration; real-time inference on edge robots may be limited to lower resolutions or fewer concurrent agents. Integration complexity is low for existing Habitat users since the API remains unchanged; however, teams need to source or generate 3DGS assets, which requires either specialized tools or training new content pipelines. The scalability story is compelling: once a scene is converted to 3DGS format, rendering becomes cheap, enabling large-scale parallel training of multiple agent copies in the same environment.
Limitations and When Not to Use This
The paper focuses on navigation tasks and does not fully explore how 3DGS rendering performs for manipulation tasks that require fine-grained depth and contact geometry—gaussians approximate surfaces but may not provide the precision needed for grasping. The 'drivable gaussian avatars' approach is not deeply detailed in the abstract; it's unclear whether the system supports real-time mocap streaming, handles occlusions well, or can simulate complex human-object interactions like humans carrying objects. The asset import pipeline assumes access to high-quality 3DGS models or the ability to train them from multi-view data, which is a barrier for scenarios where only mesh or point-cloud data exist. Finally, the paper does not address sim-to-real transfer validation—while visual fidelity improves, there's no empirical evidence that agents trained in Habitat-GS generalize better to real robots than those trained in lower-fidelity simulators, leaving a critical gap in the claim.
Research Context
Habitat-GS builds directly on Habitat-Sim (Facebook AI Research, 2019), a widely-used open-source simulator that became the standard benchmark for embodied AI. The work is motivated by recent successes in 3D Gaussian Splatting (Kerbl et al., 2023) which showed that gaussians could match NeRF quality while being 100x faster to render, opening the door for interactive applications. This paper bridges the gap between photorealistic rendering (common in VR/game engines) and embodied AI training (historically constrained to fast but low-fidelity simulation). It opens a new research direction: using differentiable rendering through gaussians to optimize agent policies jointly with scene representations, and exploring whether agents can learn from diverse human behavioral priors by training on large-scale annotated human datasets in Habitat-GS.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::