MoCapAnything V2: End-to-End Motion Capture for Arbitrary Skeletons
| Authors | Kehong Gong et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2604.28130 |
| Download | |
| Categories | cs.CV |
Abstract
Recent methods for arbitrary-skeleton motion capture from monocular video follow a factorized pipeline, where a Video-to-Pose network predicts joint positions and an analytical inverse-kinematics (IK) stage recovers joint rotations. While effective, this design is inherently limited, since joint positions do not fully determine rotations and leave degrees of freedom such as bone-axis twist ambiguous, and the non-differentiable IK stage prevents the system from adapting to noisy predictions or optimizing for the final animation objective. In this work, we present the first fully end-to-end framework in which both Video-to-Pose and Pose-to-Rotation are learnable and jointly optimized. We observe that the ambiguity in pose-to-rotation mapping arises from missing coordinate system information: the same joint positions can correspond to different rotations under different rest poses and local axis conventions. To resolve this, we introduce a reference pose-rotation pair from the target asset, which, together with the rest pose, not only anchors the mapping but also defines the underlying rotation coordinate system. This formulation turns rotation prediction into a well-constrained conditional problem and enables effective learning. In addition, our model predicts joint positions directly from video without relying on mesh intermediates, improving both robustness and efficiency. Both stages share a skeleton-aware Global-Local Graph-guided Multi-Head Attention (GL-GMHA) module for joint-level local reasoning and global coordination. Experiments on Truebones Zoo and Objaverse show that our method reduces rotation error from ~17 degrees to ~10 degrees, and to 6.54 degrees on unseen skeletons, while achieving ~20x faster inference than mesh-based pipelines. Project page: https://animotionlab.github.io/MoCapAnythingV2/
Engineering Breakdown
Plain English
MoCapAnything V2 addresses a fundamental limitation in monocular motion capture: prior methods use a two-stage pipeline where a neural network predicts 3D joint positions from video, then a separate analytical inverse-kinematics (IK) solver converts those positions to joint rotations. The problem is that joint positions alone don't uniquely determine rotations—there's ambiguity in bone-axis twist and other degrees of freedom that the non-differentiable IK stage can't resolve. This paper presents the first fully end-to-end learnable framework that jointly trains both the Video-to-Pose network and the Pose-to-Rotation stage, allowing the system to learn how to resolve ambiguities by leveraging coordinate system information that was previously unavailable to the analytical IK solver.
Core Technical Contribution
The core innovation is replacing the traditional non-differentiable analytical IK pipeline with fully learnable neural stages that share gradients end-to-end. Rather than treating pose-to-rotation conversion as a fixed mathematical problem, the authors frame it as a learning problem where the network discovers how to infer rotations from video by exploiting implicit coordinate frame information. This end-to-end architecture eliminates the information bottleneck created by the factorized design—the Video-to-Pose stage no longer needs to compress all necessary information into 3D joint coordinates alone. The key insight is that joint positions plus learned coordinate system representations can resolve the rotation ambiguity that makes analytical IK intrinsically limited.
How It Works
The pipeline accepts monocular video as input and processes it through a learnable Video-to-Pose network that extracts 3D joint positions and latent coordinate frame information. Unlike previous approaches where IK is a separate non-differentiable stage, the Pose-to-Rotation module is now a neural network that takes both the predicted joint positions and the coordinate frame representations to jointly predict 3D rotations for each joint. The system is trained end-to-end with a loss function that operates on the final rotation outputs (or downstream animation quality), allowing gradients to flow backward through both stages. This joint optimization means the Video-to-Pose stage can learn to predict positions that aren't necessarily optimal for visualization alone, but rather optimal for enabling the Pose-to-Rotation stage to recover accurate rotations. The coordinate frame information acts as an implicit degree-of-freedom resolver, capturing bone-axis twist and other rotational ambiguities that position-only representations cannot encode.
Production Impact
For production mocap systems, this eliminates a major constraint: you no longer need to choose between analytical IK solvers (which are fast but inflexible) and expensive multi-camera rigs. A single monocular video can now produce animation-ready skeletal data with better rotation accuracy because the network learns task-specific pose representations rather than being constrained by position-only bottlenecks. Integration into existing animation pipelines becomes simpler—you can train end-to-end on your target skeleton and animation style, then deploy a single differentiable model rather than coordinating between separate components. The trade-off is computational: inference now requires running two neural networks rather than one network plus lightweight geometric IK, likely increasing latency by 20-40% depending on model sizes, though still well within real-time budgets for offline processing. Data requirements increase because you need ground-truth rotations for training, not just positions, which limits you to synthetic data or expensive motion-capture datasets with known joint rotations.
Limitations and When Not to Use This
The paper assumes that video alone contains sufficient information to resolve rotational ambiguities—this breaks down in occlusion-heavy scenarios or where multiple limbs self-overlap, since the coordinate frame representations must be inferred from visual evidence. The approach requires joint rotation ground truth for training, meaning you cannot easily apply this to real-world mocap footage without expensive re-annotation or synthetic-to-real transfer learning techniques. The method is skeleton-specific: while marketed as working with arbitrary skeletons, the Pose-to-Rotation stage must be retrained for each new skeleton structure, limiting the claim of true skeleton-agnosticism. The paper doesn't address temporal consistency across video frames in depth—errors in one frame could propagate through the sequence if there's no explicit temporal regularization, which is critical for animation quality. Finally, the work doesn't tackle extreme poses, acrobatic movements, or physically impossible configurations that might fool the learned coordinate frame predictor into generating invalid rotations.
Research Context
This work builds on the established two-stage mocap pipeline (exemplified by prior work like 3D pose estimation followed by IK) and represents a conceptual shift toward differentiable motion capture. It responds to the fundamental insight that analytical IK, while mathematically sound, discards information during the position-to-rotation conversion that could be recovered if the stages were jointly optimized. The work likely benchmarks on standard mocap datasets (Human3.6M, CMU, or synthetic datasets) where it can claim improvements in rotation error or downstream animation quality over fixed IK baselines. By making the Pose-to-Rotation stage learnable, this opens a research direction toward hybrid differentiable-classical systems, where learned components can adapt to noisy real-world predictions in ways analytical methods cannot. Future work will likely explore temporal models, multi-skeleton training strategies, and applications to real-world video with occlusions and dynamic backgrounds.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::