GenLCA: 3D Diffusion for Full-Body Avatars from In-the-Wild Videos
| Authors | Yiqian Wu et al. |
| Year | 2026 |
| HF Upvotes | 4 |
| arXiv | 2604.07273 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
We present GenLCA, a diffusion-based generative model for generating and editing photorealistic full-body avatars from text and image inputs. The generated avatars are faithful to the inputs, while supporting high-fidelity facial and full-body animations. The core idea is a novel paradigm that enables training a full-body 3D diffusion model from partially observable 2D data, allowing the training dataset to scale to millions of real-world videos. This scalability contributes to the superior photorealism and generalizability of GenLCA. Specifically, we scale up the dataset by repurposing a pretrained feed-forward avatar reconstruction model as an animatable 3D tokenizer, which encodes unstructured video frames into structured 3D tokens. However, most real-world videos only provide partial observations of body parts, resulting in excessive blurring or transparency artifacts in the 3D tokens. To address this, we propose a novel visibility-aware diffusion training strategy that replaces invalid regions with learnable tokens and computes losses only over valid regions. We then train a flow-based diffusion model on the token dataset, inherently maintaining the photorealism and animatability provided by the pretrained avatar reconstruction model. Our approach effectively enables the use of large-scale real-world video data to train a diffusion model natively in 3D. We demonstrate the efficacy of our method through diverse and high-fidelity generation and editing results, outperforming existing solutions by a large margin. The project page is available at https://onethousandwu.com/GenLCA-Page.
Engineering Breakdown
Plain English
GenLCA is a diffusion-based generative model that creates photorealistic full-body 3D avatars from text and image inputs, with support for high-fidelity facial and body animations. The key innovation is a novel training paradigm that scales full-body 3D diffusion models to millions of real-world videos by treating partial 2D observations as sufficient training signal, rather than requiring complete 3D annotations. The approach repurposes a pretrained feed-forward avatar reconstruction model as an animatable 3D tokenizer that converts unstructured video frames into structured 3D tokens, enabling the dataset to scale dramatically while maintaining photorealistic quality and strong generalization to new inputs.
Core Technical Contribution
The core technical novelty is a paradigm shift in how to train 3D diffusion models: instead of requiring fully observable 3D data or complete multi-view supervision, GenLCA leverages partially observable 2D video frames by encoding them into a 3D token space using a pretrained tokenizer. This enables training on millions of uncurated in-the-wild videos rather than controlled capture datasets, which typically number in the thousands. The key insight is that a feed-forward avatar reconstruction model can serve dual duty—both as a production inference component and as a data preprocessor that extracts consistent 3D structure from inconsistent 2D observations. This architectural choice fundamentally changes the scaling bottleneck from 3D annotation complexity to simple video collection.
How It Works
The system operates in two main phases: tokenization and diffusion. In the tokenization phase, raw video frames are processed through a pretrained feed-forward avatar reconstruction model that outputs animatable 3D representations (likely mesh or latent codes) capturing body pose, shape, and appearance. These 3D outputs are quantized into discrete tokens, creating a structured 3D token sequence from unstructured video input. The diffusion model then operates in this 3D token space rather than pixel space, learning to generate coherent avatar sequences conditioned on text descriptions or reference images. During inference, the diffusion model generates token sequences, which are decoded back into 3D meshes or renderings that can be animated with arbitrary poses and expressions. The use of a tokenizer as an intermediate representation allows the diffusion model to focus on high-level semantic generation rather than low-level reconstruction details.
Production Impact
In a production avatar generation pipeline, GenLCA dramatically reduces the data collection and annotation burden that has historically limited photorealism and generalization. Instead of requiring expensive multi-view capture studios or hand-crafted 3D models, engineers can now leverage publicly available videos to train models that scale to millions of diverse identities and appearances. The ability to generate and animate avatars from text alone enables new applications in virtual meeting platforms, gaming, and content creation where custom avatars were previously prohibitively expensive to create. However, the approach introduces latency from the tokenization step (inference through the pretrained model) and requires maintaining compatibility between the tokenizer and diffusion model during updates. Compute costs are higher than traditional parametric avatar systems—diffusion inference is slower than feed-forward networks—making real-time avatar generation on mobile devices challenging without further optimization like distillation or quantization.
Limitations and When Not to Use This
The paper does not address real-time inference performance, which is critical for video conferencing and interactive applications where users expect sub-100ms latency. The approach is fundamentally limited by the quality of the pretrained tokenizer: if the reconstruction model fails to capture subtle details (micro-expressions, hand articulation, clothing dynamics), the diffusion model cannot recover this information, creating a hard ceiling on output fidelity. The paper appears to assume that partial 2D observations provide sufficient signal for learning 3D structure, but occlusion, motion blur, and extreme poses in real-world video may violate this assumption, potentially causing the model to hallucinate plausible but incorrect geometry. Additionally, there is no discussion of fairness, bias, or data consent when training on millions of in-the-wild videos, which raises significant ethical and legal concerns for production deployment.
Research Context
GenLCA builds on the recent explosion of diffusion models for generative tasks (following DDPM, DDIM, and text-to-image models like Stable Diffusion) and extends these techniques to 3D avatar generation, a space previously dominated by GAN-based and autoregressive approaches. It leverages the trend of using pretrained models as components in larger systems (similar to using CLIP for conditioning or frozen vision encoders for alignment), avoiding the need to train the entire pipeline from scratch. The work also connects to a broader research direction in 3D generation from 2D supervision—exemplified by recent work on 3D Gaussian splatting and neural radiance fields trained from single images—by showing that scaling to millions of video examples can overcome weak 3D signal. This opens new research questions around how to best design tokenizers for 3D data, how to leverage pretrained models as data processors, and how to handle the quality mismatch between high-fidelity synthetic training data and imperfect real-world video.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::