HERMES++: Toward a Unified Driving World Model for 3D Scene Understanding and Generation
| Authors | Xin Zhou et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2604.28196 |
| Download | |
| Categories | cs.CV |
Abstract
Driving world models serve as a pivotal technology for autonomous driving by simulating environmental dynamics. However, existing approaches predominantly focus on future scene generation, often overlooking comprehensive 3D scene understanding. Conversely, while Large Language Models (LLMs) demonstrate impressive reasoning capabilities, they lack the capacity to predict future geometric evolution, creating a significant disparity between semantic interpretation and physical simulation. To bridge this gap, we propose HERMES++, a unified driving world model that integrates 3D scene understanding and future geometry prediction within a single framework. Our approach addresses the distinct requirements of these tasks through synergistic designs. First, a BEV representation consolidates multi-view spatial information into a structure compatible with LLMs. Second, we introduce LLM-enhanced world queries to facilitate knowledge transfer from the understanding branch. Third, a Current-to-Future Link is designed to bridge the temporal gap, conditioning geometric evolution on semantic context. Finally, to enforce structural integrity, we employ a Joint Geometric Optimization strategy that integrates explicit geometric constraints with implicit latent regularization to align internal representations with geometry-aware priors. Extensive evaluations on multiple benchmarks validate the effectiveness of our method. HERMES++ achieves strong performance, outperforming specialist approaches in both future point cloud prediction and 3D scene understanding tasks. The model and code will be publicly released at https://github.com/H-EmbodVis/HERMESV2.
Engineering Breakdown
Plain English
HERMES++ is a unified driving world model that combines 3D scene understanding with future geometry prediction for autonomous vehicles. Most existing approaches focus either on predicting future frames or understanding current scenes, but not both together. This paper bridges that gap by using a bird's-eye-view (BEV) representation to consolidate information from multiple camera views, enabling the system to both interpret semantic scene content and simulate how 3D geometry evolves over time. The framework addresses a real limitation in autonomous driving: current systems either predict what will happen without understanding the physical world, or understand the world but can't predict its evolution.
Core Technical Contribution
The core novelty is a unified architecture that treats 3D scene understanding and future geometry prediction as synergistic rather than separate tasks. Instead of building two independent pipelines (one for semantic reasoning via LLMs, one for physical simulation), HERMES++ uses a shared BEV representation that forces the model to learn encodings that satisfy both objectives simultaneously. This multi-task design with shared representations is the key architectural insight—prior world models either focused purely on pixel-level future prediction (ignoring 3D structure) or semantic understanding (ignoring physical simulation). The BEV consolidation acts as a bottleneck that naturally enforces consistency between semantic and geometric predictions.
How It Works
The system ingests multi-view camera streams from the vehicle and fuses them into a bird's-eye-view (BEV) representation, which provides a unified 3D coordinate frame independent of camera perspective. This BEV representation is then processed through dual heads: one branch performs 3D scene understanding (detecting objects, lanes, and semantic structure), while another branch predicts how that 3D geometry will evolve frame-by-frame into the future. The shared BEV encoding ensures that the model cannot make predictions that contradict its scene understanding—if it understands an object's position and velocity, those must be consistent with its future predictions. The multi-task learning objective combines losses from both branches, forcing the model to learn representations that encode both instantaneous 3D structure and temporal dynamics. This differs from prior approaches that either treat future prediction as a pixel-generation task (decoupled from 3D reasoning) or generate semantic descriptions without grounding in physics.
Production Impact
For autonomous driving systems, this approach solves the integration problem between perception and prediction pipelines. Currently, engineers maintain separate modules: a perception stack that detects and classifies objects, and a prediction stack that forecasts trajectories—these often make conflicting decisions about object locations and velocities. HERMES++ unifies these by forcing geometric and semantic consistency in a single forward pass, reducing latency compared to running cascaded pipelines and eliminating conflicts between perception and prediction outputs. The BEV representation is hardware-friendly (2D tensors rather than 3D voxels) so inference compute is manageable for automotive edge devices. The trade-off is increased training data requirements (need both semantic annotations and trajectory ground truth) and more complex loss engineering to balance multiple objectives. Integration requires multi-view camera calibration and careful tuning of loss weights between the understanding and prediction heads.
Limitations and When Not to Use This
The paper's abstract doesn't specify performance metrics, which makes it unclear whether the unified approach actually outperforms well-tuned separate models or simply provides architectural convenience. The approach assumes reliable multi-view camera inputs and calibration, which can be fragile in adverse weather or when cameras are dirty—failure modes for BEV fusion aren't discussed. It's unclear how the model handles occlusions or out-of-distribution scenarios (e.g., novel object types not in training data) where semantic understanding and geometric prediction might diverge. The method also likely requires substantial labeled data with both 3D geometric ground truth and semantic labels, which is expensive to collect and may not scale to rare edge cases important for safety-critical driving.
Research Context
This work builds on a decade of BEV perception research in autonomous driving and recent advances in multi-task learning for vision. It acknowledges the recent surge of LLMs being applied to driving (reasoning about scenarios) but identifies a gap: LLMs lack grounding in physical simulation. The paper fits into a broader trend of unifying perception and prediction rather than treating them as separate stages, similar to end-to-end driving approaches but with explicit 3D scene structure rather than direct control outputs. The research opens up future work on incorporating language reasoning into BEV-based prediction, leveraging LLM semantic understanding to constrain geometric predictions toward physically plausible futures.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::