3D-ReGen: A Unified 3D Geometry Regeneration Framework
| Authors | Geon Yeong Park et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2604.28134 |
| Download | |
| Categories | cs.CV |
Abstract
We consider the problem of regenerating 3D objects from 2D images and initial 3D shapes. Most 3D generators operate in a one-shot fashion, converting text or images to a 3D object with limited controllability. We introduce instead 3D-ReGen, a 3D regenerator that is conditioned on an initial 3D shape. This conceptually simple formulation allows us to support numerous useful tasks, including 3D enhancement, reconstruction, and editing. 3D-ReGen uses a new conditioning mechanism based on VecSet, which allows the regenerator to update or improve the input geometry with consistent fine-grained details. 3D-ReGen learns a widely applicable regeneration prior from off-the-shelf 3D datasets via self-supervised pretext tasks and augmentations, without additional annotations. We evaluate both the geometric consistency and fine-grained quality of 3D-ReGen, achieving state-of-the-art performance in controllable 3D generation across several tasks.
Engineering Breakdown
Plain English
3D-ReGen is a generative framework that takes a 2D image and an initial 3D shape as input, then regenerates an improved 3D object with better geometry and fine-grained details. Unlike one-shot 3D generators that produce geometry from text or images alone, this approach conditions the regeneration on an existing 3D shape, enabling multiple downstream tasks: enhancement, reconstruction, and editing. The key innovation is a VecSet-based conditioning mechanism that allows the model to selectively update input geometry while maintaining consistency. The system learns from off-the-shelf 3D datasets using self-supervised pretext tasks without requiring manual annotations, making it broadly applicable across different 3D object categories.
Core Technical Contribution
The core innovation is a unified conditioning architecture using VecSet that enables 3D shapes to be progressively refined rather than generated from scratch. This differs fundamentally from prior one-shot generators by treating 3D regeneration as a conditional refinement problem where an initial shape serves as both input and structural anchor. The VecSet mechanism operates on point-cloud or mesh representations, allowing the model to understand which geometric elements to preserve, modify, or enhance based on the 2D image guidance. The self-supervised pretraining approach learns a regeneration prior without paired annotations, using only augmentations and pretext tasks on raw 3D data, which is a significant departure from supervised 3D generation methods that typically require synthetic or manually-labeled datasets.
How It Works
The system takes two inputs: a 2D image (source of visual guidance) and an initial 3D shape (geometric anchor). These are encoded separately—the image through a standard vision encoder and the 3D shape through a geometry encoder that produces a VecSet representation, which is an unordered collection of geometric features that can be queried and updated. The VecSet conditioning mechanism acts as a structured bridge between the 2D and 3D modalities, allowing the decoder to iteratively refine or regenerate the 3D geometry. At each decoding step, the model decides whether to keep, modify, or add geometric primitives (vertices, faces, or features) based on attention between the image features and current geometry state. The training objective is self-supervised, using augmentations of the same 3D shape (rotation, occlusion, noise) as positive pairs and different shapes as negatives, similar to contrastive learning. The output is a regenerated 3D object with improved surface detail, cleaner geometry, and consistency with both the input shape and 2D image.
Production Impact
This approach directly solves the problem of iterative 3D asset refinement in production pipelines—rather than regenerating geometry from scratch (which can be unstable or lose important details), you can now improve existing models while preserving critical structural information. For 3D content creation workflows (game dev, CAD, digital art), this enables semi-automated enhancement where artists provide rough geometry and 2D references, and the system refines the 3D model automatically. The self-supervised pretraining requirement is modest compared to supervised alternatives—you only need raw 3D datasets without annotations—making it feasible to fine-tune on domain-specific data (architectural models, mechanical parts, character assets). However, the conditioning mechanism adds latency during inference due to iterative refinement steps, and memory requirements scale with 3D resolution; you'd need to benchmark whether this fits your real-time or batch processing constraints. Integration complexity is moderate: you need a 3D representation pipeline (mesh or point-cloud processing) and a 2D vision encoder, but the overall architecture is more modular than end-to-end 3D generators since the initial shape provides strong inductive bias.
Limitations and When Not to Use This
The paper assumes the initial 3D shape is topologically or geometrically similar to the target—if the input shape is drastically different from what the 2D image suggests, the conditioning mechanism may struggle to perform meaningful refinement. There's no discussion of how the method handles category shift (training on cars, generalizing to chairs) or extreme viewpoint changes in the 2D image; these generalization failures could limit real-world applicability. The evaluation metrics and datasets are not fully detailed in the abstract, so we don't know if this was tested on standard benchmarks (ShapeNet, PartNet) or whether quantitative comparisons to prior work validate the claimed advantages. Computational cost during inference (iterative refinement steps) and training (self-supervised pretraining on large 3D corpora) is not analyzed, which matters for production deployment. The method likely requires careful tuning of the VecSet representation (how many features, dimensionality) for different 3D categories, suggesting it may not be truly universal despite the framing.
Research Context
This work builds on recent advances in 3D generative models (NeRF-based and diffusion-based generators) while pivoting toward a conditional refinement paradigm rather than unconditional generation. It's motivated by the success of text-to-image conditioned generation (like CLIP or Stable Diffusion) but extends this to the 3D-to-3D domain with multi-modal (2D image + 3D shape) conditioning. The VecSet concept appears to be inspired by set-based representations in computer vision (Sets of Local Features, PointNet-style architectures) adapted for fine-grained geometry manipulation. This research direction opens up new opportunities in iterative 3D editing, human-in-the-loop refinement, and task-specific geometry optimization—areas where one-shot generators have been limited. The self-supervised learning strategy also aligns with broader trends in reducing annotation burden for 3D vision, similar to recent progress in self-supervised point-cloud learning.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::