OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation
| Authors | Jinghui Lu et al. |
| Year | 2026 |
| HF Upvotes | 85 |
| arXiv | 2604.18486 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Chain-of-Thought (CoT) reasoning has become a powerful driver of trajectory prediction in VLA-based autonomous driving, yet its autoregressive nature imposes a latency cost that is prohibitive for real-time deployment. Latent CoT methods attempt to close this gap by compressing reasoning into continuous hidden states, but consistently fall short of their explicit counterparts. We suggest that this is due to purely linguistic latent representations compressing a symbolic abstraction of the world, rather than the causal dynamics that actually govern driving. Thus, we present OneVL (One-step latent reasoning and planning with Vision-Language explanations), a unified VLA and World Model framework that routes reasoning through compact latent tokens supervised by dual auxiliary decoders. Alongside a language decoder that reconstructs text CoT, we introduce a visual world model decoder that predicts future-frame tokens, forcing the latent space to internalize the causal dynamics of road geometry, agent motion, and environmental change. A three-stage training pipeline progressively aligns these latents with trajectory, language, and visual objectives, ensuring stable joint optimization. At inference, the auxiliary decoders are discarded and all latent tokens are prefilled in a single parallel pass, matching the speed of answer-only prediction. Across four benchmarks, OneVL becomes the first latent CoT method to surpass explicit CoT, delivering state-of-the-art accuracy at answer-only latency, and providing direct evidence that tighter compression, when guided in both language and world-model supervision, produces more generalizable representations than verbose token-by-token reasoning. Project Page: https://xiaomi-embodied-intelligence.github.io/OneVL
Engineering Breakdown
Plain English
This paper addresses a critical bottleneck in autonomous driving: Chain-of-Thought reasoning improves trajectory prediction accuracy but is too slow for real-time deployment because it requires sequential token generation. The authors propose OneVL, which compresses reasoning into compact latent tokens supervised by dual decoder heads (a world model and a language model), eliminating the autoregressive bottleneck while maintaining reasoning quality. The key insight is that purely linguistic latent representations fail because they abstract away the causal dynamics of driving; instead, they route reasoning through vision-grounded latent states that capture actual world physics. This enables one-step prediction with reasoning comparable to multi-step Chain-of-Thought approaches.
Core Technical Contribution
The core novelty is a unified Vision-Language-Action (VLA) and World Model framework that decouples reasoning from generation. Instead of generating reasoning tokens autoregressively (which are then decoded into actions), OneVL compresses reasoning into a single latent bottleneck token trained with dual supervision: one decoder reconstructs the visual world state (world model loss), and another generates explanatory text (language modeling loss). This dual auxiliary decoder design forces the latent representation to capture causal, physics-grounded reasoning rather than mere linguistic abstraction. The architectural contribution is using vision as a grounding signal for latent reasoning—the latent tokens must be interpretable both as world dynamics and as language, creating an implicit constraint that prevents pure symbol compression.
How It Works
The input consists of visual observations (camera frames) and the current driving state (position, velocity, etc.). The vision encoder processes images and feeds them alongside the state into a unified VLA encoder that produces a sequence of latent reasoning tokens. These tokens are compressed through a bottleneck mechanism into a compact latent representation designed for one-step planning. Two auxiliary decoders operate in parallel: (1) a world model decoder that reconstructs the next visual frame and predicted world state, and (2) a language model decoder that generates a natural language explanation of the reasoning. The final action decoder (e.g., steering angle, acceleration) uses both the latent tokens and the world model predictions to generate the control command. During inference, the bottleneck ensures only a minimal latent representation flows through, reducing latency from multiple forward passes to essentially one.
Production Impact
For autonomous driving systems, this directly addresses the latency-accuracy trade-off that currently forces engineers to choose between slow-but-accurate CoT reasoning and fast-but-brittle direct prediction. Adopting OneVL would eliminate the need to run multi-step token generation in the loop, reducing end-to-end latency from potentially 500-1000ms (for 10-20 CoT steps) to a single forward pass (~50-100ms), compatible with real-time control loops. The dual decoder supervision adds computational overhead during training (two auxiliary loss terms, two decoders), but inference cost remains fixed—slightly higher than direct prediction but orders of magnitude lower than autoregressive CoT. Integration challenges include: (1) retraining the world model decoder for each new deployment environment, (2) ensuring the latent bottleneck doesn't collapse during training, and (3) validating that implicit world model supervision generalizes to unseen driving scenarios better than explicit reasoning.
Limitations and When Not to Use This
The paper relies on the assumption that dual supervision (world model + language) is sufficient to learn causal driving dynamics, but vision-only world models are notoriously fragile in edge cases (occlusions, adversarial scenarios, distributional shifts). The approach requires labeled data for next-frame prediction and natural language explanations, which is expensive to scale; the paper does not discuss data efficiency or performance on small datasets. Latent bottleneck methods are known to suffer from posterior collapse during training (where the latent space becomes uninformative), and the paper does not provide ablation studies on the bottleneck width or regularization strategies to prevent this. Finally, the evaluation is limited to trajectory prediction; real autonomous driving requires closed-loop interaction, where prediction errors compound—this work does not demonstrate performance in actual vehicle control loops or long-horizon rollouts.
Research Context
This work builds directly on latent CoT research (prior work showing that compressing reasoning into hidden states can speed up inference) and world models for planning (e.g., Dreamer, PlaNet), but is the first to combine both in a unified VLA framework. It improves upon purely linguistic latent reasoning by introducing visual grounding as a regularizer, extending recent insights from multimodal learning that vision can constrain reasoning. The paper contributes to the broader trajectory of making transformer-based planning tractable for real-time robotics and driving—a key bottleneck as the field moves from offline benchmarks (nuScenes, CARLA) to deployment-grade systems. It opens research directions in multi-modal auxiliary losses, architectural design for latent reasoning, and the role of world models as implicit reasoning supervisors.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::