PhyMotion: Structured 3D Motion Reward for Physics-Grounded Human Video Generation
:::info Stub — Full Engineering Breakdown Coming This paper has a linked code implementation and was featured on Hugging Face Papers with 6 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Yidong Huang et al. |
| Year | 2026 |
| HF Upvotes | 6 |
| arXiv | 2605.14269 |
| Download | |
| Code | https://github.com/h6kplus/PhyMotion |
Abstract
Generating realistic human motion is a central yet unsolved challenge in video generation. While reinforcement learning (RL)-based post-training has driven recent gains in general video quality, extending it to human motion remains bottlenecked by a reward signal that cannot reliably score motion realism. Existing video rewards primarily rely on 2D perceptual signals, without explicitly modeling the 3D body state, contact, and dynamics underlying articulated human motion, and often assign high scores to videos with floating bodies or physically implausible movements. To address this, we propose PhyMotion, a structured, fine-grained motion reward that grounds recovered 3D human trajectories in a physics simulator and evaluates motion quality along multiple dimensions of physical feasibility. Concretely, we recover SMPL body meshes from generated videos, retarget them onto a humanoid in the MuJoCo physics simulator, and evaluate the resulting motion along three axes: kinematic plausibility, contact and balance consistency, and dynamic feasibility. Each component provides a continuous and interpretable signal tied to a specific aspect of motion quality, allowing the reward to capture which aspects of motion are physically correct or violated. Experiments show that PhyMotion achieves stronger correlation with human judgments than existing reward formulations. These gains carry over to RL-based post-training, where optimizing PhyMotion leads to larger and more consistent improvements than optimizing existing rewards, improving motion realism across both autoregressive and bidirectional video generators under both automatic metrics and blind human evaluation (+68 Elo gain). Ablations show that the three axes provide complementary supervision signals, while the reward preserves overall video generation quality with only modest training overhead.
Engineering Breakdown
Plain English
This paper addresses a critical limitation in AI video generation: existing reward signals for training motion-generation models can't reliably distinguish between physically realistic and physically implausible human movement (like floating bodies or impossible joint rotations). PhyMotion proposes a structured reward function that extracts 3D body pose from generated videos, simulates the motion in a physics engine, and scores videos based on multiple physics-grounded metrics—contact with ground, joint constraints, dynamics feasibility—rather than relying on 2D visual perception alone.
Key Engineering Insight
The core insight is that 2D perceptual rewards are fundamentally insufficient for human motion tasks because they can't detect physical impossibilities that are invisible in 2D space (like legs passing through floors or joints bending backwards). By grounding the reward in a 3D physics simulator, you get an objective function that actually penalizes physics violations rather than rewarding them as visually plausible artifacts.
Why It Matters for Engineers
If you're building production video generation systems for human motion (avatar synthesis, motion capture alternatives, video editing), reward signal quality directly determines model output quality. Using a physics-grounded reward instead of generic video quality metrics should substantially reduce the costly manual filtering work needed to remove physically broken outputs—a real pain point in deploying these systems today.
Research Context
Video generation with RL post-training has improved general quality, but human motion remained problematic because existing rewards (like clip-based perceptual scorers) optimize for visual appearance, not physical validity. PhyMotion advances this by introducing domain-specific structure into the reward—physics simulation isn't new, but embedding it into the RL training loop for video generation is. This enables training pipelines that directly optimize for realistic articulated motion rather than treating it as a side effect of visual realism.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::