TAIHRI: Task-Aware 3D Human Keypoints Localization for Close-Range Human-Robot Interaction
| Authors | Ao Li et al. |
| Year | 2026 |
| HF Upvotes | 2 |
| arXiv | 2604.08921 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Accurate 3D human keypoints localization is a critical technology enabling robots to achieve natural and safe physical interaction with users. Conventional 3D human keypoints estimation methods primarily focus on the whole-body reconstruction quality relative to the root joint. However, in practical human-robot interaction (HRI) scenarios, robots are more concerned with the precise metric-scale spatial localization of task-relevant body parts under the egocentric camera 3D coordinate. We propose TAIHRI, the first Vision-Language Model (VLM) tailored for close-range HRI perception, capable of understanding users' motion commands and directing the robot's attention to the most task-relevant keypoints. By quantizing 3D keypoints into a finite interaction space, TAIHRI precisely localize the 3D spatial coordinates of critical body parts by 2D keypoint reasoning via next token prediction, and seamlessly adapt to downstream tasks such as natural language control or global space human mesh recovery. Experiments on egocentric interaction benchmarks demonstrate that TAIHRI achieves superior estimation accuracy for task-critical body parts. We believe TAIHRI opens new research avenues in the field of embodied human-robot interaction. Code is available at: https://github.com/Tencent/TAIHRI.
Engineering Breakdown
Plain English
This paper introduces TAIHRI, a Vision-Language Model designed specifically for close-range human-robot interaction where robots need to understand human motion commands and precisely locate task-relevant body parts in 3D space from an egocentric camera view. Unlike conventional 3D human pose estimation methods that focus on whole-body reconstruction relative to a root joint, TAIHRI tackles the practical HRI problem of metric-scale spatial localization of specific body parts that matter for the robot's task. The key innovation is quantizing 3D keypoints into a finite interaction space, allowing the model to ground its spatial understanding in the robot's coordinate frame rather than just reconstructing human pose. This approach enables robots to achieve more natural and safer physical interaction by attending to the exact location of task-relevant keypoints in real-world metric space.
Core Technical Contribution
The core novelty is adapting Vision-Language Models to the egocentric, task-centric requirements of human-robot interaction rather than applying general-purpose pose estimation. TAIHRI combines VLM capabilities with spatial quantization to map 3D keypoint locations into a discrete interaction space that the robot can reason about and act upon. Unlike previous work that treats keypoint localization as a camera-relative reconstruction problem, this approach explicitly grounds keypoints in metric space relevant to the robot's manipulation tasks. The paper is the first to systematically combine language understanding (for motion commands) with spatially-grounded 3D keypoint detection (for precise metric localization) in a single unified model for HRI.
How It Works
The system takes an egocentric RGB image from the robot's camera and a natural language instruction about which body parts to attend to as inputs. The VLM encodes the image and language prompt to understand both the visual scene and the task-relevant semantics (which keypoints matter for this interaction). Rather than regressing continuous 3D coordinates, TAIHRI quantizes the 3D space around the robot into discrete bins or regions, converting the keypoint localization problem into a classification task over these spatial cells. The model outputs discrete spatial locations for each relevant keypoint, which get decoded back into metric 3D coordinates in the robot's frame. This quantization approach provides two benefits: it grounds predictions in real physical space the robot operates in, and it reduces the complexity of learning fine-grained continuous coordinate regression. The language component allows the model to selectively activate attention on task-relevant keypoints rather than estimating all 17+ body keypoints every time.
Production Impact
For robotics engineers building manipulation systems, TAIHRI solves a critical gap: conventional pose estimators don't provide robot-frame spatial accuracy needed for close-contact tasks like dressing assistance, physical therapy, or collaborative assembly. Replacing a generic pose estimator with TAIHRI would improve metric localization accuracy for task-relevant joints (hands, arms, torso) while reducing computational overhead by selectively predicting only relevant keypoints. The language grounding enables more intuitive task specification—robots can parse verbal or written task descriptions to determine which body parts to track with high precision. The quantization strategy reduces latency since discrete classification is faster than regression and requires less memory, improving real-time control feasibility. Trade-offs include the need for task-specific training data (egocentric HRI scenarios are less abundant than in-the-wild pose data), higher annotation complexity since you need metric ground truth in robot frame rather than just keypoint visibility, and potential accuracy loss from discretization if the quantization granularity doesn't match your specific manipulation workspace.
Limitations and When Not to Use This
The paper assumes close-range interaction scenarios with clear egocentric views—it likely struggles when the user is far from the robot's camera, partially occluded, or the robot's camera is not optimally positioned. The quantization approach introduces discretization error that grows with bin size; for delicate tasks requiring sub-centimeter accuracy, this could be problematic unless the quantization space is made very fine-grained, increasing compute. The approach requires collecting task-specific egocentric training data paired with metric ground truth in the robot's frame, which is expensive and limits generalization across different robot platforms and workspace geometries. The paper doesn't discuss how the model handles multiple people, dynamic backgrounds, or domain shift when deployed on different robot hardware—these are common failure modes in real production systems. There's also an open question about how well the language grounding generalizes to novel task descriptions or to users with different body proportions and movement patterns than the training distribution.
Research Context
This work builds on recent progress in Vision-Language Models (like CLIP and its extensions) that can ground natural language in visual understanding, adapting them for the robotics domain where spatial grounding must be metric and egocentric. It improves upon conventional 3D human pose estimation benchmarks (COCO, Human3.6M) by reframing the evaluation around robot-relevant metrics: precision of task-critical joints in robot frame coordinates rather than mean per-joint position error. The paper opens a research direction at the intersection of embodied AI, egocentric perception, and human-robot collaboration—future work could explore how to scale this to full-body task understanding, multi-person scenarios, or cross-robot generalization. It also suggests a broader pattern: many robotic perception problems may benefit from task-specific adaptation of foundation models rather than off-the-shelf application of general vision or pose estimation models.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::