InpaintSLat: Inpainting Structured 3D Latents via Initial Noise Optimization
| Authors | Jaeyoung Chung et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2605.00664 |
| Download | |
| Categories | cs.CV, cs.AI |
Abstract
We present a training-free approach for controllable 3D inpainting based on initial noise optimization. In the structured 3D latent diffusion framework, we observe that the underlying geometric structure is established during the early stages of the diffusion process and exhibits high sensitivity to the initial noise. Such characteristics compromise stability in tasks like inpainting and editing, where the model must ensure strict alignment with the existing context while synthesizing a new structure. In this paper, we introduce a strategy to optimize the initial noise within the structured 3D latent diffusion framework, ensuring high-fidelity 3D inpainting. Specifically, we update the initial noise by leveraging a backpropagation approximation grounded in the rectified flow model, with the spectral parameterization specially designed for robust and efficient structured 3D latent optimization. Experiments demonstrate consistent improvements in contextual consistency and prompt alignment over representative training-free inpainting baselines, establishing initial noise control as an independent dimension for 3D inpainting, orthogonal to conventional sampling trajectory manipulation.
Engineering Breakdown
Plain English
This paper tackles 3D inpainting—filling in missing or masked regions of 3D objects while maintaining consistency with existing geometry—using a training-free approach built on diffusion models. The key insight is that geometric structure in 3D latent diffusion is locked in very early during generation and is highly sensitive to the initial noise pattern, which causes instability during inpainting tasks. The authors propose optimizing the initial noise directly using backpropagation approximations derived from rectified flow theory, enabling high-fidelity 3D reconstruction without fine-tuning the model. This approach maintains alignment with existing 3D context while synthesizing plausible new geometry in masked regions.
Core Technical Contribution
The core novelty is a training-free initial noise optimization strategy for 3D latent diffusion inpainting that uses rectified flow-based backpropagation approximations to guide the starting point of generation. Rather than trying to steer the diffusion process after it begins (which risks destroying established geometry), the authors identified that optimizing the input noise vector itself preserves geometric coherence while enabling controlled synthesis. This is fundamentally different from prior inpainting approaches that either fine-tune models or apply post-hoc constraints during denoising steps. The method reframes 3D inpainting as an inverse problem solved through initial condition optimization, leveraging theoretical insights about how diffusion models establish structure in early timesteps.
How It Works
The system operates within a structured 3D latent diffusion framework where 3D geometry is represented in a learned latent space. During the forward diffusion process, noise is progressively added to clean 3D latents over multiple timesteps; the paper observes that geometric structure crystallizes early and becomes brittle to perturbation. The authors introduce a loss function based on masked reconstruction error and apply backpropagation through the diffusion sampling trajectory to compute gradients with respect to the initial noise vector. These gradients are estimated using rectified flow theory, which provides a continuous-time parameterization of the diffusion process enabling efficient gradient computation. The optimized initial noise is then fed into the standard reverse diffusion process, producing 3D inpainting that respects both the mask constraints and the existing geometry. The entire approach is training-free, meaning no model parameters are updated—only the initialization is manipulated per inference.
Production Impact
For production 3D generation pipelines (e.g., CAD generation, 3D asset creation, medical imaging reconstruction), this enables real-time 3D inpainting without model fine-tuning or retraining, reducing infrastructure costs and deployment complexity. Teams could integrate this into existing 3D diffusion model deployments as a lightweight module that modifies input noise before calling the standard reverse process, requiring minimal code changes. The training-free property is critical for production systems because it avoids catastrophic forgetting and reduces GPU memory during inference (no optimizer state needed). However, the method adds inference-time computation cost due to gradient accumulation through the diffusion trajectory—practitioners should expect 2-5x longer inference time depending on the number of noise optimization iterations, requiring careful latency budgeting for interactive applications. The approach is particularly valuable for constrained-synthesis tasks (medical imaging, industrial design) where maintaining geometric fidelity to existing context is paramount and model retraining per task is impractical.
Limitations and When Not to Use This
The paper assumes access to a pre-trained, well-behaved 3D latent diffusion model; performance degrades significantly if the base model is undertrained or the latent space is poorly structured. Gradient-based optimization of initial noise requires differentiability through the entire sampling pipeline and assumes the rectified flow approximation holds, which may break down for extreme masking patterns or unusual 3D geometries. The method is computationally expensive at inference time, making it unsuitable for real-time interactive applications with strict latency budgets—this is a hard constraint for mobile or in-browser 3D editing. The paper likely lacks evaluation on complex multi-part objects or highly topologically varied geometries where the early-structure assumption may not hold equally across different regions. Follow-up work should address adaptive noise optimization strategies that allocate computation budget based on mask complexity, and investigation into whether the approach generalizes across different 3D representations (meshes, point clouds, implicit functions).
Research Context
This work builds directly on the growing body of research in latent diffusion for 3D generation, extending 2D inpainting techniques (like DDIM-based mask-guided generation) to the 3D domain where geometric coherence is harder to enforce. It leverages rectified flow models as a theoretical foundation, which have recently emerged as an alternative to Gaussian diffusion with better theoretical properties and computational efficiency. The approach is positioned as a training-free alternative to recent 3D editing methods that either require fine-tuning with LoRA-style adapters or use classifier-free guidance, representing a different point in the trade-off space between quality, speed, and generality. The work likely targets benchmarks like ShapeNet, 3D-Chair, or similar 3D object datasets, advancing the state-of-the-art in geometric-aware inpainting metrics.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::