Mimic Intent, Not Just Trajectories
| Authors | Renming Huang et al. |
| Year | 2026 |
| HF Upvotes | 14 |
| arXiv | 2602.08602 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
While imitation learning (IL) has achieved impressive success in dexterous manipulation through generative modeling and pretraining, state-of-the-art approaches like Vision-Language-Action (VLA) models still struggle with adaptation to environmental changes and skill transfer. We argue this stems from mimicking raw trajectories without understanding the underlying intent. To address this, we propose explicitly disentangling behavior intent from execution details in end-2-end IL: Mimic Intent, Not just Trajectories(MINT). We achieve this via multi-scale frequency-space tokenization, which enforces a spectral decomposition of action chunk representation. We learn action tokens with a multi-scale coarse-to-fine structure, and force the coarsest token to capture low-frequency global structure and finer tokens to encode high-frequency details. This yields an abstract Intent token that facilitates planning and transfer, and multi-scale Execution tokens that enable precise adaptation to environmental dynamics. Building on this hierarchy, our policy generates trajectories through next-scale autoregression, performing progressive intent-to-execution reasoning, thus boosting learning efficiency and generalization. Crucially, this disentanglement enables one-shot transfer of skills, by simply injecting the Intent token from a demonstration into the autoregressive generation process. Experiments on several manipulation benchmarks and on a real robot demonstrate state-of-the-art success rates, superior inference efficiency, robust generalization against disturbances, and effective one-shot transfer.
Engineering Breakdown
Plain English
This paper addresses a fundamental problem in imitation learning for robotic manipulation: existing Vision-Language-Action models and trajectory-based approaches fail to adapt to new environments or transfer skills because they learn to copy raw action sequences without understanding the underlying intent behind those actions. The authors propose MINT (Mimic Intent, Not just Trajectories), which uses multi-scale frequency-space tokenization to explicitly separate high-level behavioral intent (captured in low-frequency components) from low-level execution details (captured in high-frequency components). This spectral decomposition forces the model to learn what the human was trying to do, rather than just replicating joint movements, enabling better generalization and skill transfer across different environmental conditions and task variants.
Core Technical Contribution
The core innovation is a multi-scale coarse-to-fine tokenization structure that operates in frequency space rather than raw action space. By decomposing action chunks into spectral components, the authors enforce a hierarchical representation where the coarsest token is constrained to capture only low-frequency global structure—essentially the intent or goal—while finer tokens handle execution details. This is different from prior VLA models and generative imitation learning approaches that treat actions as monolithic tokens or continuous vectors without explicit structure. The key insight is that intentional behavior has a clear frequency signature: high-level goals (low frequency) are distinct from fine motor adjustments (high frequency), and this natural decomposition can be leveraged to improve generalization and transfer.
How It Works
The system takes trajectory data as input and applies a frequency-space decomposition to action chunks—converting the raw action sequences into their frequency components using techniques similar to Fourier or wavelet analysis. The action representation is then tokenized at multiple scales: a coarse token captures the dominant low-frequency components (representing intent), and progressively finer tokens capture mid and high-frequency details. During training, the model learns to reconstruct full action trajectories while being explicitly constrained so that the coarse tokens only encode low-frequency information. The architecture likely combines a frequency decomposer, a multi-scale tokenizer, and a reconstruction module that synthesizes fine-grained actions from intent tokens. At inference time, the model can generate actions by first predicting intent from vision and language inputs, then conditionally generating execution details—enabling the model to preserve intent while adapting execution to new environments.
Production Impact
In production robotic systems, this approach would significantly improve adaptation to deployment-time variations: a robot trained on one setup could better generalize to different lighting, camera angles, object positions, or even hardware variations because it's learned the intent rather than memorizing specific trajectories. For companies building robot fleets, this reduces the retraining burden—you can collect demonstrations in one lab and deploy with higher success rates in varied real-world conditions. The trade-off is increased model complexity: you need frequency decomposition and multi-scale tokenization infrastructure, which adds computational overhead during both training and inference (though not prohibitively—standard frequency transforms are O(n log n)). Integration requires changing how action data is preprocessed and how action tokens are structured in your IL pipeline, but the benefit is substantial for dexterous manipulation tasks where human demonstrations are expensive to collect.
Limitations and When Not to Use This
The paper doesn't address how to choose the frequency cutoff that separates intent from execution—this threshold may be task-dependent and require empirical tuning. The approach assumes that behavioral intent actually manifests as low-frequency structure in action space, which may not hold for tasks with rapid, high-frequency intentional movements (e.g., rapid drumming or fast striking). The evaluation likely focuses on specific manipulation benchmarks; it's unclear how well the method scales to locomotion, navigation, or other domains with different action dynamics. Additionally, the paper doesn't discuss what happens when human demonstrations themselves contain noise or inconsistency in how they execute the same intent—the frequency decomposition might struggle to reliably separate true intent from demonstration artifacts.
Research Context
This work builds on recent successes in scaling imitation learning through vision-language models and generative action modeling, but specifically targets the known brittleness of these approaches when faced with distribution shift. It draws from classical signal processing (frequency decomposition) applied to a modern problem in embodied AI. The research contributes to a growing body of work questioning whether trajectory mimicry is the right learning target—similar to how recent work in RL has emphasized learning reward structures over policy cloning. The paper likely benchmarks on standard dexterous manipulation datasets (LIBERO, RLBench, or similar) and opens a new research direction: how to structure learned representations to disentangle abstraction levels in imitation learning, which could influence future architectures for robot learning and potentially cross-domain transfer problems.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::