Learning Versatile Humanoid Manipulation with Touch Dreaming
| Authors | Yaru Niu et al. |
| Year | 2026 |
| HF Upvotes | 3 |
| arXiv | 2604.13015 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Humanoid robots promise general-purpose assistance, yet real-world humanoid loco-manipulation remains challenging because it requires whole-body stability, dexterous hands, and contact-aware perception under frequent contact changes. In this work, we study dexterous, contact-rich humanoid loco-manipulation. We first develop an RL-based whole-body controller that provides stable lower-body and torso execution during complex manipulation. Built on this controller, we develop a whole-body humanoid data collection system that combines VR-based teleoperation with human-to-humanoid motion mapping, enabling efficient collection of real-world demonstrations. We then propose Humanoid Transformer with Touch Dreaming (HTD), a multimodal encoder--decoder Transformer that models touch as a core modality alongside multi-view vision and proprioception. HTD is trained in a single stage with behavioral cloning augmented by touch dreaming: in addition to predicting action chunks, the policy predicts future hand-joint forces and future tactile latents, encouraging the shared Transformer trunk to learn contact-aware representations for dexterous interaction. Across five contact-rich tasks, Insert-T, Book Organization, Towel Folding, Cat Litter Scooping, and Tea Serving, HTD achieves a 90.9% relative improvement in average success rate over the stronger baseline. Ablation results further show that latent-space tactile prediction is more effective than raw tactile prediction, yielding a 30% relative gain in success rate. These results demonstrate that combining robust whole-body execution, scalable humanoid data collection, and predictive touch-centered learning enables versatile, high-dexterity humanoid manipulation in the real world. Project webpage: humanoid-touch-dream.github.io.
Engineering Breakdown
Plain English
This paper tackles a fundamental challenge in robotics: enabling humanoid robots to perform complex manipulation tasks that require simultaneous whole-body stability, dexterous hand control, and awareness of contact forces. The authors built a complete system starting with an RL-based whole-body controller for locomotion stability, then created an efficient data collection pipeline using VR teleoperation and human-to-robot motion mapping. The core contribution is HTD (Humanoid Transformer with Touch Dreaming), a multimodal transformer architecture that treats tactile feedback as a primary input modality alongside vision, enabling the robot to learn versatile manipulation skills from real-world demonstrations.
Core Technical Contribution
The key novelty is elevating touch/tactile feedback from a secondary signal to a core modality in multimodal transformer learning for humanoid control. Rather than treating touch as an auxiliary input, HTD explicitly models contact dynamics and forces as a first-class feature in an encoder-decoder transformer, enabling the model to reason about contact state changes during complex loco-manipulation. This is combined with a practical data collection system that bridges the sim-to-real gap through VR teleoperation and kinematic retargeting, making it feasible to gather diverse manipulation demonstrations on real hardware.
How It Works
The system operates in three integrated stages. First, an RL-trained whole-body controller maintains stability of the lower body and torso while manipulating, serving as a foundation for safe physical interaction. Second, the data collection pipeline uses VR controllers worn by a human operator combined with inverse kinematics mapping to retarget their movements onto the humanoid's kinematic structure, capturing diverse manipulation demonstrations with natural contact interactions. Third, HTD processes these demonstrations as a sequence-to-sequence problem: the encoder processes multimodal observations (visual features, proprioceptive state, and crucially, tactile sensor readings from the robot's hands) and the decoder generates action sequences that preserve contact awareness by explicitly modeling touch dynamics across the entire manipulation trajectory.
Production Impact
For teams building real humanoid systems, this approach directly addresses three blocking problems: stability during contact-heavy tasks (solved by the whole-body controller), efficient real-world data collection (solved by the VR+retargeting pipeline), and learning from contact-aware demonstrations (solved by HTD's touch-centric architecture). The production benefit is that you can now train humanoid manipulation skills without large-scale simulation or extensive hand-coded contact models — instead, collecting data becomes manageable through teleoperation, and the transformer learns contact dynamics implicitly. Trade-offs include: significant compute for training multimodal transformers, hardware cost for tactile sensors across the hand, VR equipment and operator training overhead for data collection, and latency concerns if real-time control is needed (transformers require careful optimization for on-robot inference).
Limitations and When Not to Use This
The paper assumes access to high-quality tactile sensors across the hand, which is still relatively rare and expensive in production humanoids; systems without distributed touch sensing cannot use this approach directly. It does not address the challenge of distributing learned skills across different hardware platforms or hands with different tactile layouts, limiting generalization. The approach requires substantial human operator effort for VR teleoperation, making it unclear whether this scales to learning hundreds of diverse skills. Additionally, the paper does not thoroughly evaluate failure modes during contact transitions (e.g., slipping, sudden force spikes) or provide quantitative benchmarks on contact prediction accuracy versus baseline approaches, leaving open questions about robustness in adversarial scenarios.
Research Context
This work builds on the convergence of three research threads: whole-body RL control for humanoids (extending prior locomotion controllers to manipulation), multimodal transformers for robotics (following the success of vision-language models adapted to embodied AI), and tactile learning which has historically been underexplored compared to vision. It advances the state-of-the-art in dexterous manipulation by showing that touch can be leveraged as a primary learning signal rather than auxiliary feedback, similar to how recent work in vision-language learning elevated language to a first-class modality. The practical data collection methodology also addresses a persistent gap between simulation-based RL research and real-world deployment, positioning this as part of the emerging agenda around efficient real-robot learning.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::