HumanNet: Scaling Human-centric Video Learning to One Million Hours
:::info Stub — Full Engineering Breakdown Coming This paper has a linked code implementation and was featured on Hugging Face Papers with 24 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Yufan Deng & Daquan Zhou |
| Year | 2026 |
| HF Upvotes | 24 |
| arXiv | 2605.06747 |
| Download | |
| Code | https://github.com/DAGroup-PKU/HumanNet |
Abstract
Progress in embodied intelligence increasingly depends on scalable data infrastructure. While vision and language have scaled with internet corpora, learning physical interaction remains constrained by the lack of large, diverse, and richly annotated human activity data. We present HumanNet, a one-million-hour human-centric video corpus that captures how humans interact with the physical world at scale. HumanNet spans both first-person and third-person perspectives and covers fine-grained activities, human-object interactions, tool use, and long-horizon behaviors across diverse real-world environments. Beyond raw video, the dataset provides interaction-centric annotations, including captions, motion descriptions, and hand and body-related signals, enabling motion-aware and interaction-aware learning. Beyond scale, HumanNet introduces a systematic data curation paradigm for embodied learning, where human-centric filtering, temporal structuring, viewpoint diversity, and annotation enrichment are treated as first-class design principles. This design transforms unstructured internet video into a scalable substrate for representation learning, activity understanding, motion generation, and human-to-robot transfer. We conduct a first-step validation on the value of this design through controlled vision-language-action ablation: under a fixed set of validation data, continued training from the Qwen VLM model with 1000 hours of egocentric video drawn from HumanNet surpasses the continued training with 100 hours of real-robot data from Magic Cobot, indicating that egocentric human video could be a scalable and cost-effective substitute for robot data. By building this project, we aim to explore the opportunity to scale embodied foundation models using human-centric videos, rather than relying solely on robot-specific data.
Engineering Breakdown
Plain English
HumanNet is a one-million-hour video dataset focused on human physical interactions with the world, captured from both first-person and third-person perspectives. Beyond raw video, it includes rich annotations like motion descriptions, hand/body signals, and interaction captions—essentially building a massive corpus for training embodied AI systems that need to understand how humans actually manipulate objects and perform complex tasks.
Key Engineering Insight
The critical engineering win here is shifting from unstructured video to interaction-centric annotations (hand signals, motion vectors, object interactions). This structured metadata makes the dataset immediately useful for training models that need to predict and understand physical manipulation, not just classify video frames.
Why It Matters for Engineers
Building embodied AI (robots, VR systems, AR applications) has hit a hard wall: there's no large-scale training data for human-object interaction. Teams currently scrape YouTube or build small custom datasets—expensive and limited. HumanNet removes this bottleneck, letting you train interaction models the way vision models trained on ImageNet or language models on web-scale text.
Research Context
Vision and NLP scaled by capturing internet-scale data; embodied AI remained stuck with small labeled datasets because human interaction videos are hard to annotate meaningfully. This paper solves that by creating the first million-hour corpus with interaction-aware labels, enabling the same scaling playbook for physical AI that worked for vision and language.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::