PhysForge: Generating Physics-Grounded 3D Assets for Interactive Virtual World
:::info Stub — Full Engineering Breakdown Coming This paper was featured on Hugging Face Daily Papers on 2026-05-06 with 34 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Yunhan Yang et al. |
| Year | 2026 |
| HF Upvotes | 34 |
| arXiv | 2605.05163 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Synthesizing physics-grounded 3D assets is a critical bottleneck for interactive virtual worlds and embodied AI. Existing methods predominantly focus on static geometry, overlooking the functional properties essential for interaction. We propose that interactive asset generation must be rooted in functional logic and hierarchical physics. To bridge this gap, we introduce PhysForge, a decoupled two-stage framework supported by PhysDB, a large-scale dataset of 150,000 assets with four-tier physical annotations. First, a VLM acts as a "physical architect" to plan a "Hierarchical Physical Blueprint" defining material, functional, and kinematic constraints. Second, a physics-grounded diffusion model realizes this blueprint by synthesizing high-fidelity geometry alongside precise kinematic parameters via a novel KineVoxel Injection (KVI) mechanism. Experiments demonstrate that PhysForge produces functionally plausible, simulation-ready assets, providing a robust data engine for interactive 3D content and embodied agents.
Engineering Breakdown
Plain English
PhysForge is a two-stage framework for generating 3D assets that actually work in interactive virtual worlds—not just look good statically. The system uses a VLM to plan physical constraints (materials, functionality, kinematics) in a structured blueprint, then a physics-aware diffusion model generates the actual 3D geometry that respects those constraints. They built PhysDB, a dataset of 150,000 assets with four-tier physics annotations, to train on real functional properties rather than just geometry.
Key Engineering Insight
Decoupling planning (what physical properties should this asset have?) from generation (now synthesize geometry that respects those properties) lets you ground generative models in functional constraints before synthesis happens. This mirrors how production systems need explicit specifications before execution, not implicit emergence during generation.
Why It Matters for Engineers
Interactive worlds and embodied AI need assets that don't just render—they need to behave correctly. Chairs should be sittable, doors should open with correct pivot joints, objects should have realistic mass distribution. Current generative models produce geometrically plausible but functionally broken assets. This directly impacts sim-to-real transfer and reduces the manual engineering tax of fixing broken physics after generation.
Research Context
Prior work treated asset generation as a pure geometry problem (NeRF, diffusion models for 3D shapes). PhysForge advances this by treating functionality as a first-class requirement, not an afterthought. The hierarchical blueprint approach and large-scale annotated dataset enable moving from 'generate something that looks 3D' to 'generate something that physically works,' unlocking practical use in game engines, robotics simulators, and metaverse platforms that can't tolerate broken physics.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::