HY-Embodied-0.5: Embodied Foundation Models for Real-World Agents
| Authors | Tencent Robotics X et al. |
| Year | 2026 |
| HF Upvotes | 159 |
| arXiv | 2604.07430 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
We introduce HY-Embodied-0.5, a family of foundation models specifically designed for real-world embodied agents. To bridge the gap between general Vision-Language Models (VLMs) and the demands of embodied agents, our models are developed to enhance the core capabilities required by embodied intelligence: spatial and temporal visual perception, alongside advanced embodied reasoning for prediction, interaction, and planning. The HY-Embodied-0.5 suite comprises two primary variants: an efficient model with 2B activated parameters designed for edge deployment, and a powerful model with 32B activated parameters targeted for complex reasoning. To support the fine-grained visual perception essential for embodied tasks, we adopt a Mixture-of-Transformers (MoT) architecture to enable modality-specific computing. By incorporating latent tokens, this design effectively enhances the perceptual representation of the models. To improve reasoning capabilities, we introduce an iterative, self-evolving post-training paradigm. Furthermore, we employ on-policy distillation to transfer the advanced capabilities of the large model to the smaller variant, thereby maximizing the performance potential of the compact model. Extensive evaluations across 22 benchmarks, spanning visual perception, spatial reasoning, and embodied understanding, demonstrate the effectiveness of our approach. Our MoT-2B model outperforms similarly sized state-of-the-art models on 16 benchmarks, while the 32B variant achieves performance comparable to frontier models such as Gemini 3.0 Pro. In downstream robot control experiments, we leverage our robust VLM foundation to train an effective Vision-Language-Action (VLA) model, achieving compelling results in real-world physical evaluations. Code and models are open-sourced at https://github.com/Tencent-Hunyuan/HY-Embodied.
Engineering Breakdown
Plain English
HY-Embodied-0.5 is a family of foundation models built specifically for robots and embodied AI agents that need to understand and interact with the physical world. The paper addresses a key gap: general Vision-Language Models (VLMs) aren't optimized for the spatial-temporal reasoning, prediction, and planning that embodied agents require. The authors release two model variants—a 2B parameter efficient model for edge deployment on robots and a 32B parameter model for complex reasoning tasks—using a Mixture-of-Transformers architecture to handle different visual modalities separately. This represents a significant shift from treating embodied AI as a direct application of generic VLMs to building purpose-built models with the right inductive biases for real-world robot control and scene understanding.
Core Technical Contribution
The core novelty is introducing Mixture-of-Transformers (MoT) architecture specifically for embodied agents, which allows different transformer experts to specialize in different modalities and sensory streams needed for robotics (e.g., separate handling of egocentric vision, depth maps, object segmentation). Rather than forcing embodied reasoning through a generic VLM architecture, the authors redesigned the foundation model from the ground up to optimize for the spatial and temporal visual perception that robots actually need—understanding 3D scene structure, predicting object trajectories, and planning motor actions. The two-tier model family (2B efficient / 32B powerful) with activated parameters rather than total parameters suggests they're using sparsity and mixture-of-experts techniques to scale computational requirements based on deployment context. This is fundamentally different from prior approaches that fine-tune general VLMs for robotics; instead, this treats embodied intelligence as a first-class modeling problem requiring architectural innovations.
How It Works
The input to HY-Embodied-0.5 consists of multi-modal egocentric observations from a robot (RGB images, depth data, proprioceptive feedback, and task instructions in natural language). The Mixture-of-Transformers architecture routes these inputs to specialized transformer experts based on modality—for instance, one expert branch might process spatial geometric information from depth and camera streams while another handles semantic and temporal reasoning across frames. The model learns to predict future states of the environment (next-frame prediction), understand object interactions (what happens when the robot grasps this object), and generate action sequences (planning)—these embodied reasoning tasks are built into the training objective rather than bolted on afterward. The efficient 2B variant uses parameter sharing and selective routing to keep memory and compute low for edge deployment on actual robots, while the 32B variant can maintain all expert parameters for scenarios where it runs on a workstation or cloud backend. The output is either a continuous action distribution (for motion planning), a predicted future state (for world model tasks), or structured reasoning tokens that explain the decision-making process.
Production Impact
For engineers deploying robots in real environments, this changes the foundation model strategy fundamentally. Instead of starting with a general VLM like CLIP or LLaVA and bolting robotics-specific adapters on top, you can now use HY-Embodied-0.5 directly as your visual encoder and reasoning backbone, getting better spatial understanding and temporal prediction out of the box. The 2B efficient variant is deployable directly on robot edge hardware (NVIDIA Jetson, mobile accelerators) which eliminates cloud round-trip latency for perception—critical for safety and responsiveness in real-time control. The architecture's mixture-of-experts design means you can selectively activate experts during inference based on the current task (e.g., only activate grasping experts if performing manipulation), reducing compute per inference. However, the trade-off is that you're now locked into the HY-Embodied ecosystem: fine-tuning requires understanding the MoT routing mechanism, you need embodied-specific data for transfer learning rather than generic vision datasets, and integration requires robotics-specific harnesses for action output, state encoding, and temporal batching that differ from standard VLM APIs.
Limitations and When Not to Use This
The paper assumes access to sufficient embodied training data (robot trajectories, interaction sequences, world models) which is expensive and domain-specific to collect at scale—it's unclear how well the models transfer across robot morphologies, environments, or task distributions outside their training domain. The Mixture-of-Transformers architecture adds complexity to serving and optimization; it's not clear how well standard inference optimizations (quantization, distillation, compiler fusion) work with dynamic routing, which could limit practical edge deployment gains. The paper doesn't clearly specify what 'activated parameters' means (is it mixture-of-experts sparsity, parameter sharing, or something else), making it hard to assess whether the 2B/32B comparison is apples-to-apples with other efficient models. Real-world embodied tasks often have failure modes from distribution shift (novel objects, new environments), and the paper doesn't address robustness testing or how the model degrades when observations fall outside the training distribution—crucial for safe deployment. There's also a gap between prediction/planning in simulation versus actual motor control; the paper focuses on understanding and prediction but doesn't clearly demonstrate end-to-end closed-loop control on real robots with the model as the primary decision-maker.
Research Context
This work builds directly on the embodied AI / robotics foundation model trend started by models like RT-1, RT-2, and OpenVLA, but makes a architectural departure by designing for multimodal specialists rather than a monolithic transformer. It improves on generic VLM-based approaches (CLIP, LLaVA applied to robotics) by recognizing that spatial-temporal reasoning for embodiment requires different inductive biases than natural language understanding. The research opens up a new direction in building modality-specialist foundation models via mixture-of-experts, which could influence how we design models for other multi-sensor domains (autonomous vehicles, UAVs, industrial perception). By releasing two model scales (2B efficient and 32B capable), the authors are providing a reference architecture for the embodied AI community similar to how Llama-2 / Llama-3 scaled up language model availability for industry—expect this to become a baseline for robotics research and deployment much like CLIP became for vision-language tasks.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::