Repurposing 3D Generative Model for Autoregressive Layout Generation
| Authors | Haoran Feng et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2604.16299 |
| Download | |
| Categories | cs.CV |
Abstract
We introduce LaviGen, a framework that repurposes 3D generative models for 3D layout generation. Unlike previous methods that infer object layouts from textual descriptions, LaviGen operates directly in the native 3D space, formulating layout generation as an autoregressive process that explicitly models geometric relations and physical constraints among objects, producing coherent and physically plausible 3D scenes. To further enhance this process, we propose an adapted 3D diffusion model that integrates scene, object, and instruction information and employs a dual-guidance self-rollout distillation mechanism to improve efficiency and spatial accuracy. Extensive experiments on the LayoutVLM benchmark show LaviGen achieves superior 3D layout generation performance, with 19% higher physical plausibility than the state of the art and 65% faster computation. Our code is publicly available at https://github.com/fenghora/LaviGen.
Engineering Breakdown
Plain English
LaviGen repurposes 3D generative models to solve the problem of generating coherent 3D scene layouts directly from text descriptions, treating it as an autoregressive sequence generation problem rather than translating text into 2D instructions. The key innovation is operating natively in 3D space while explicitly modeling geometric relationships and physical constraints between objects, ensuring generated scenes are both spatially coherent and physically plausible. The authors augment this with an adapted 3D diffusion model that uses dual-guidance self-rollout distillation for improved efficiency and spatial accuracy. On the LayoutVLM benchmark, LaviGen achieves 19% higher physical plausibility compared to prior methods—a substantial improvement for a task where physical validity is critical for downstream applications.
Core Technical Contribution
The core contribution is reformulating 3D layout generation as an autoregressive process that operates directly in 3D space rather than intermediate representations (like text or 2D layouts). Unlike prior work that infers object positions from text descriptions through indirect means, LaviGen explicitly models geometric and physical constraints as part of the generation process itself, making these relationships learnable and enforceable. The secondary contribution is the adapted 3D diffusion model with dual-guidance self-rollout distillation—a mechanism that improves both computational efficiency and spatial accuracy by combining scene context, object attributes, and user instructions into a unified guidance signal. This repurposing of 3D generative model components (diffusion-based architecture) for a new task (layout generation) demonstrates how existing model families can be adapted with task-specific mechanisms.
How It Works
The system takes a text description of a desired 3D scene and processes it through an autoregressive generation pipeline that sequentially places objects one at a time in 3D space. Each step generates the next object's position, orientation, and scale while conditioning on previously placed objects and a global scene context vector derived from the text description and scene constraints. The 3D diffusion model component denoises spatial configurations iteratively, starting from noise and progressively refining object positions to satisfy both the text description and physical plausibility constraints (e.g., objects should not interpenetrate, should rest on surfaces appropriately). The dual-guidance mechanism injects three types of information simultaneously: the overall scene semantics (from the text), individual object properties, and explicit user instruction embeddings, allowing the model to balance descriptive fidelity with physical realism. Self-rollout distillation involves the model generating complete layouts, evaluating them for validity, and using valid examples as training targets—a form of reinforcement learning through generated trajectory ranking. The output is a fully specified 3D layout with all objects positioned in a unified 3D coordinate system, ready for rendering or downstream scene understanding tasks.
Production Impact
For product teams building 3D scene generation tools (game engines, interior design software, architectural visualization), LaviGen enables end-to-end generation from natural language descriptions in a single forward pass, eliminating the need for intermediate 2D-to-3D lifting or separate constraint solvers. The 19% improvement in physical plausibility means fewer post-processing steps and manual corrections by artists or designers, reducing iteration cycles significantly. Integration would require: (1) training or fine-tuning on your domain-specific layout dataset (LayoutVLM benchmark shows this is achievable), (2) careful engineering of the constraint model to match domain-specific rules (e.g., furniture-specific physics may differ from robot manipulation), and (3) handling variable scene complexity—the autoregressive approach's latency scales with the number of objects, so scenes with 50+ objects may require optimization. The efficiency gains from self-rollout distillation mean the model is competitive with or faster than prior iterative refinement approaches, making real-time or interactive applications feasible if properly engineered.
Limitations and When Not to Use This
The paper achieves superior results specifically on the LayoutVLM benchmark, but generalization to other domain-specific layouts (e.g., outdoor scenes, large-scale urban environments, highly constrained manufacturing layouts) is unexplored and likely requires substantial retraining or fine-tuning. The autoregressive approach has a fundamental sequential dependency: errors in early object placement can cascade to later objects, and there is no mechanism to revisit or globally optimize object positions—this could be problematic for scenes where initial placement decisions constrain later choices heavily. Physical constraint modeling relies on a learned diffusion component, which may fail in novel scenarios with complex multi-object interactions (e.g., stacked objects, load-bearing requirements) that fall outside the training distribution. The paper does not discuss how to handle continuous refinement of user preferences mid-generation or partial scene completion, limiting interactivity compared to iterative design tools. Finally, computational cost for high-resolution 3D diffusion and the data requirements for training on large-scale 3D scene datasets are not thoroughly analyzed—this could be a blocker for resource-constrained production environments.
Research Context
This work builds on recent advances in 3D generative models (particularly diffusion-based approaches) and multimodal scene understanding, extending methods from image-text models into the 3D domain. It directly addresses limitations of prior layout generation methods that operated on 2D projections or used text-as-an-intermediate representation, instead following the trend of native 3D reasoning in generative models. The improvement on the LayoutVLM benchmark (a recently introduced benchmark for evaluating 3D layout generation from language) demonstrates the value of the autoregressive formulation and dual-guidance distillation for this specific task. This opens research directions in: (1) extending autoregressive 3D generation to other scene-understanding tasks, (2) exploring global optimization over locally-greedy sequential decisions, and (3) scaling 3D diffusion models to handle more complex scenes and longer sequences.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::