ReImagine: Rethinking Controllable High-Quality Human Video Generation via Image-First Synthesis
| Authors | Zhengwentai Sun et al. |
| Year | 2026 |
| HF Upvotes | 3 |
| arXiv | 2604.19720 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Human video generation remains challenging due to the difficulty of jointly modeling human appearance, motion, and camera viewpoint under limited multi-view data. Existing methods often address these factors separately, resulting in limited controllability or reduced visual quality. We revisit this problem from an image-first perspective, where high-quality human appearance is learned via image generation and used as a prior for video synthesis, decoupling appearance modeling from temporal consistency. We propose a pose- and viewpoint-controllable pipeline that combines a pretrained image backbone with SMPL-X-based motion guidance, together with a training-free temporal refinement stage based on a pretrained video diffusion model. Our method produces high-quality, temporally consistent videos under diverse poses and viewpoints. We also release a canonical human dataset and an auxiliary model for compositional human image synthesis. Code and data are publicly available at https://github.com/Taited/ReImagine.
Engineering Breakdown
Plain English
This paper tackles the hard problem of generating realistic human videos by separating appearance modeling from motion control. The authors propose a pipeline that first learns high-quality human appearance through image generation, then uses that as a foundation for video synthesis guided by SMPL-X pose and viewpoint parameters. A key innovation is a training-free temporal refinement stage using a pretrained video diffusion model to ensure frames stay consistent over time. The approach produces temporally coherent, controllable human videos without requiring extensive multi-view training data, addressing a major bottleneck in human video generation.
Core Technical Contribution
The core novelty is the image-first decoupling strategy: instead of jointly modeling appearance, motion, and viewpoint (which creates an under-constrained optimization problem), the authors leverage a pretrained image generation backbone to handle appearance in isolation, then use that learned prior as an anchor for video generation. SMPL-X body model integration provides explicit, interpretable control over pose and camera viewpoint without manual annotation. The training-free temporal refinement using a diffusion model is elegant—it avoids expensive per-video fine-tuning by reusing pretrained video models, making the pipeline practical for deployment. This separation of concerns is fundamentally different from end-to-end approaches that try to predict all factors jointly from limited supervision.
How It Works
The pipeline operates in three stages. First, a pretrained image generation model (likely a latent diffusion model) receives appearance conditioning and generates high-quality reference frames of the human. Second, these appearance embeddings flow into a video synthesis component guided by SMPL-X skeleton data—the body model provides 3D pose parameters and camera intrinsics that are injected as structural priors into the generation process, giving explicit control over viewpoint and motion. Third, a pretrained video diffusion model runs in inference-only mode to refine temporal consistency across frames without any training—this leverages the temporal coherence learned by large-scale video models without per-video optimization. The input is a target pose sequence and camera parameters; the output is a coherent video with consistent appearance and controllable motion.
Production Impact
For teams building avatar systems or human-centric content creation tools, this approach dramatically reduces the data and compute burden compared to training end-to-end models on multi-view video datasets. You can now generate reasonable results using only pretrained image and video models plus SMPL-X pose annotations—no need to collect expensive multi-camera video rigs. Latency is reasonable: image generation is a few seconds per frame, but the inference-only temporal refinement avoids the optimization loop that makes some video generation pipelines slow. Trade-offs include: the approach is still slower than real-time (likely 30-60 seconds per 10-second video on a high-end GPU), and visual quality depends heavily on the quality of pretrained models you start with. Integration is straightforward—plug in your SMPL-X pose estimator, feed frames to the pipeline, run temporal refinement, and output video.
Limitations and When Not to Use This
The paper assumes access to good pretrained image and video diffusion models, which may not exist for all domains (e.g., full-body motion in extreme lighting or non-human anatomies). The SMPL-X model itself has known failure modes with complex hand gestures and extreme poses, which will propagate downstream. The training-free temporal refinement is clever but not guaranteed to fix temporal artifacts if the underlying appearance frames are inconsistent—there are limits to what post-hoc diffusion can correct. The abstract is truncated, so critical details about failure cases, quantitative benchmarks, and comparison to concurrent methods are missing; real production validation would require access to the full paper and reproducible experiments on standard benchmarks.
Research Context
This work sits at the intersection of recent advances in diffusion models for image and video generation, and the long-standing problem of human pose transfer and avatar synthesis. It builds on prior work using SMPL/SMPL-X as structural priors (like neural body models) but innovates by decoupling appearance from motion at a systems level rather than within a single model. The image-first strategy echoes recent trends in conditional generation (e.g., ControlNet, which uses detectable features like pose as guidance), but applies it to the full video synthesis pipeline. This likely advances benchmarks on human motion transfer datasets or video re-rendering tasks, and opens a research direction toward modular video generation where appearance, motion, and viewpoint are learned and controlled independently.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::