SeeThrough3D: Occlusion Aware 3D Control in Text-to-Image Generation
| Authors | Vaibhav Agrawal et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2602.23359 |
| Download | |
| Categories | cs.CV, cs.AI |
Abstract
We identify occlusion reasoning as a fundamental yet overlooked aspect for 3D layout-conditioned generation. It is essential for synthesizing partially occluded objects with depth-consistent geometry and scale. While existing methods can generate realistic scenes that follow input layouts, they often fail to model precise inter-object occlusions. We propose SeeThrough3D, a model for 3D layout conditioned generation that explicitly models occlusions. We introduce an occlusion-aware 3D scene representation (OSCR), where objects are depicted as translucent 3D boxes placed within a virtual environment and rendered from desired camera viewpoint. The transparency encodes hidden object regions, enabling the model to reason about occlusions, while the rendered viewpoint provides explicit camera control during generation. We condition a pretrained flow based text-to-image image generation model by introducing a set of visual tokens derived from our rendered 3D representation. Furthermore, we apply masked self-attention to accurately bind each object bounding box to its corresponding textual description, enabling accurate generation of multiple objects without object attribute mixing. To train the model, we construct a synthetic dataset with diverse multi-object scenes with strong inter-object occlusions. SeeThrough3D generalizes effectively to unseen object categories and enables precise 3D layout control with realistic occlusions and consistent camera control.
Engineering Breakdown
Plain English
This paper addresses a critical gap in 3D layout-conditioned image generation: existing models struggle with occlusion reasoning—understanding which objects should be hidden behind others and how their geometry changes when partially occluded. The authors propose SeeThrough3D, which introduces an Occlusion-aware 3D Scene Representation (OSCR) where objects are modeled as translucent 3D boxes in a virtual environment, with transparency values encoding hidden regions. The key innovation is rendering these boxes from a desired camera viewpoint before feeding them to the generation model, allowing the system to explicitly reason about depth ordering, inter-object occlusions, and geometrically consistent partial visibility. This enables synthesis of realistic, physically plausible scenes where occluded objects maintain correct scale, position, and depth relationships.
Core Technical Contribution
The core novelty is the Occlusion-aware 3D Scene Representation (OSCR)—a structured 3D intermediate representation that makes occlusion reasoning explicit and differentiable within text-to-image generation. Rather than treating layout as flat 2D bounding boxes or unstructured scene graphs, OSCR represents each object as a translucent 3D box with transparency encoding occlusion depth, then renders this scene from the target camera view as a 3D-aware conditioning signal. This bridges the gap between 3D geometry reasoning and 2D image generation by forcing the diffusion model to observe and condition on occlusion patterns before synthesis. Prior work either ignored occlusions entirely or handled them implicitly through post-processing; SeeThrough3D makes them a first-class component of the generation pipeline.
How It Works
The pipeline works as follows: (1) User provides a text prompt, a 3D scene layout (object positions, sizes, camera viewpoint), and optionally semantic segmentation masks. (2) The OSCR module converts this layout into a set of translucent 3D bounding boxes positioned in world space; objects further from camera or behind others are rendered with reduced opacity to encode occlusion. (3) These boxes are rendered from the specified camera viewpoint using a differentiable renderer, producing a single RGB image where transparency and depth ordering are visually encoded. (4) This rendered OSCR image is passed as a conditioning input (alongside text embeddings) to a diffusion-based text-to-image model, likely finetuned or augmented with cross-attention mechanisms to attend to both text and the 3D-aware spatial conditioning. (5) The model iteratively denoises from Gaussian noise to produce a final image that respects the 3D layout, maintains occlusion consistency, and aligns with the text description. The transparency values act as a soft constraint that guides the model to synthesize appropriate partial visibility and geometric consistency without hard masking.
Production Impact
For production systems, this enables a new class of applications: architecture visualization tools that can reliably render partially occluded furniture or structural elements; e-commerce product visualization where overlapping items maintain realistic depth relationships; 3D scene design interfaces where layout changes instantly propagate to photorealistic renders. Integration into existing text-to-image pipelines would require two key changes: (1) adding a 3D scene parser/layout annotation step upstream (slight latency overhead for scene understanding, ~100-200ms depending on annotation method), and (2) finetuning or adapter-tuning the base diffusion model on occlusion-labeled datasets (non-trivial data collection and training cost). Compute-wise, rendering OSCR for each sample is cheap (~10-50ms on CPU), but the finetuned diffusion model may have similar per-sample generation cost as baseline (20-60 seconds on GPU for high quality). The main trade-off: you gain precise 3D-aware control and occlusion correctness but lose some of the zero-shot flexibility of layout-agnostic text-to-image models, and you need clean 3D layout annotations.
Limitations and When Not to Use This
The approach assumes users can provide or the system can accurately infer precise 3D layouts (positions, sizes, camera intrinsics), which is non-trivial in practice—noisy or ambiguous annotations will degrade results. OSCR representation assumes axis-aligned boxes, which limits expressiveness for non-convex objects, complex articulated bodies, or organic shapes; thin objects or fine geometric details may not render clearly enough to guide generation. The method relies on training or finetuning the diffusion model on occluded scene data, introducing data scarcity risk if such labeled datasets are limited; generalization to novel object categories or viewpoints unseen during training is unclear. Additionally, the paper does not address dynamic occlusions (moving objects) or temporal consistency in video generation, limiting applicability to animated or interactive scenarios.
Research Context
This work builds on a decade of 3D-aware image generation research (e.g., NeRF-based conditioning, 3D scene graphs for layout control) but identifies occlusion as the missing critical piece that prior methods glossed over. It sits at the intersection of 3D scene understanding, diffusion models, and spatial control—extending recent 3D layout-conditioned generation work (such as spatial control in Stable Diffusion) by adding explicit depth reasoning. The paper likely benchmarks against methods like GLIGEN, LayoutControl, or similar 3D/spatial conditioning baselines, demonstrating improvements in occlusion correctness, depth consistency, and object scale plausibility. It opens a research direction toward tighter coupling of 3D geometry and 2D synthesis, potentially inspiring follow-up work on transparent/semi-transparent object handling, multi-view consistency, and camera-aware generation.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::