PhyCo: Learning Controllable Physical Priors for Generative Motion
| Authors | Sriram Narayanan et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2604.28169 |
| Download | |
| Categories | cs.CV, cs.AI, cs.LG |
Abstract
Modern video diffusion models excel at appearance synthesis but still struggle with physical consistency: objects drift, collisions lack realistic rebound, and material responses seldom match their underlying properties. We present PhyCo, a framework that introduces continuous, interpretable, and physically grounded control into video generation. Our approach integrates three key components: (i) a large-scale dataset of over 100K photorealistic simulation videos where friction, restitution, deformation, and force are systematically varied across diverse scenarios; (ii) physics-supervised fine-tuning of a pretrained diffusion model using a ControlNet conditioned on pixel-aligned physical property maps; and (iii) VLM-guided reward optimization, where a fine-tuned vision-language model evaluates generated videos with targeted physics queries and provides differentiable feedback. This combination enables a generative model to produce physically consistent and controllable outputs through variations in physical attributes-without any simulator or geometry reconstruction at inference. On the Physics-IQ benchmark, PhyCo significantly improves physical realism over strong baselines, and human studies confirm clearer and more faithful control over physical attributes. Our results demonstrate a scalable path toward physically consistent, controllable generative video models that generalize beyond synthetic training environments.
Engineering Breakdown
Plain English
PhyCo is a framework that fixes a major weakness in video diffusion models: they generate visually plausible videos but break physical laws—objects drift through space, collisions don't bounce realistically, and materials don't deform properly. The authors solve this by creating a 100K+ dataset of photorealistic simulation videos with systematically varied physics parameters (friction, restitution, deformation, force), then fine-tuning a diffusion model using ControlNet conditioned on pixel-aligned physical property maps, and optimizing with a VLM-based reward function. The result is a generative model that produces videos where physics actually works: objects collide and bounce correctly, materials respond to forces in realistic ways, and motion remains consistent throughout the sequence.
Core Technical Contribution
The core novelty is a three-stage pipeline that makes video diffusion models physically grounded without requiring physics simulators at inference time. First, the authors created a massive curated dataset of simulation videos where physical parameters are explicitly varied and annotated—this is the first large-scale, physics-labeled video dataset designed specifically for learning controllable motion generation. Second, they developed a ControlNet-based conditioning mechanism that takes pixel-aligned physical property maps (friction, mass, elasticity) as inputs, allowing fine-grained control over generated motion. Third, they introduced VLM-guided reward optimization where a vision-language model trained on physics scenarios scores generated videos for physical plausibility, creating a feedback loop that doesn't require ground-truth physics simulation. This combination of explicit physical supervision, interpretable conditioning, and learned reward signals is novel—prior work either uses full physics simulation (slow, expensive) or ignores physics entirely (unrealistic).
How It Works
The system operates in three stages. Stage one: the input to PhyCo is a prompt (text description or image) plus optional physical property maps specifying friction coefficients, restitution values, material deformations, and applied forces, all registered to pixel coordinates in the scene. Stage two: these inputs are encoded into a ControlNet module that conditions a pretrained video diffusion model during the denoising process—at each diffusion timestep, the model uses both the semantic prompt and the physical constraints to guide which pixels should move and how. Stage three: during training, a fine-tuned vision-language model evaluates generated videos on physical plausibility (does that ball bounce with the right energy? does friction match the material?), producing scalar reward signals that are backpropagated through the diffusion model to improve physics compliance. The output is a video that matches both the semantic content (person walking, object falling) and the physical constraints (correct bounce height, material-appropriate deformation). Inference takes a standard diffusion reverse process (typically 50-100 steps) with physics-aware conditioning, adding minimal latency over uncontrolled diffusion.
Production Impact
For teams building video generation systems, PhyCo solves a critical credibility problem: generated videos for training robotics models, simulating physical scenarios, or creating VFX will now maintain physical consistency, making them viable as synthetic training data. In practice, this means you could generate 10,000 videos of a robotic arm grasping objects with physically accurate contact and friction, rather than spending weeks acquiring real footage or running expensive physics simulators. The trade-off is significant: you need to label or automatically extract physical property maps for your scenes (friction, restitution, mass), which adds annotation overhead; the model requires fine-tuning on domain-specific simulation data (the 100K dataset may not cover your use case); and inference latency is comparable to standard diffusion but requires conditioning data preparation. For organizations already running physics simulation pipelines, PhyCo could replace some simulation workloads—generating plausible variations on physically consistent scenarios much faster than running RigidBody engines. For VFX or game development, this enables artists to control physics in generated videos directly through interpretable parameters rather than expensive re-renders or manual tweaking.
Limitations and When Not to Use This
PhyCo assumes you have access to or can generate labeled simulation data with explicit physical parameters—the approach won't work well for scenarios where ground-truth physics properties are unknown (complex fabrics, fluid interactions, soft-body deformations). The VLM-based reward model is only as good as its training data; if it's trained primarily on rigid-body mechanics, it will fail to evaluate soft-body or fluid dynamics correctly, and the fine-tuning may overfit to those failure modes. The method also requires pixel-aligned physical property maps as input, which is a significant usability constraint—in practice, annotating friction and restitution for every pixel in a complex scene is tedious and error-prone. Finally, the paper doesn't address temporal consistency over very long sequences (does physics remain stable across 10-second videos?) or complex multi-body interactions with many contact points, suggesting the approach may break down on more realistic, chaotic scenarios. The compute cost of VLM-guided optimization and repeated fine-tuning on different datasets is also not fully characterized.
Research Context
PhyCo builds directly on recent advances in diffusion-based video generation (Models like Runway, Pika) and ControlNet conditioning (Lvmin Zhang et al., 2023), which showed that conditioning diffusion models on spatial controls produces high-quality, semantically aligned outputs. It's also informed by the success of large-scale simulation datasets for physics understanding and the emerging use of vision-language models as learned reward functions (similar to recent work in RLHF-based vision tasks). The paper opens a research direction around 'physics-aware generative models'—the next steps would likely be extending this to fluid dynamics, cloth simulation, and long-horizon planning scenarios, and reducing the annotation burden through self-supervised physical property extraction. It also positions simulation datasets as a new commodity in the generative AI pipeline, similar to how ImageNet enabled supervised vision, suggesting that large annotated physics simulation repos will become standard research infrastructure.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::