The Compression Gap: Why Discrete Tokenization Limits Vision-Language-Action Model Scaling
| Authors | Takuya Shiba |
| Year | 2026 |
| Field | AI / ML |
| arXiv | 2604.03191 |
| Download | |
| Categories | cs.RO, cs.CV, cs.LG |
Abstract
Scaling Vision-Language-Action (VLA) models by upgrading the vision encoder is expected to improve downstream manipulation performance--as it does in vision-language modeling. We show that this expectation fails when actions are represented as discrete tokens, and explain why through an information-theoretic principle we call the Compression Gap: in any visuomotor pipeline, scaling behavior is governed by the location of the tightest information bottleneck. When actions are continuous (e.g., Diffusion Policy), the vision encoder is the binding constraint, and upgrading it directly improves performance. When actions are discretized through a fixed-capacity codebook (e.g., OAT), the codebook becomes the binding constraint, and encoder improvements cannot propagate past it--regardless of how rich the upstream representation is. We validate this principle on the LIBERO benchmark with three lines of evidence: a factorial experiment showing that encoder upgrades improve Diffusion Policy by over 21 percentage points while OAT gains are substantially attenuated across model scales; an encoder quality gradient across four encoders confirming that Diffusion Policy tracks encoder quality monotonically while OAT remains flat; and a codebook size experiment demonstrating that relaxing codebook capacity partially recovers encoder sensitivity, providing causal evidence for the bottleneck hypothesis. Our findings reveal that scaling in Physical AI requires identifying where information bottlenecks lie in the pipeline, rather than uniformly increasing model or data size.
Engineering Breakdown
Plain English
This paper investigates why scaling vision encoders in Vision-Language-Action (VLA) models—which works well in traditional vision-language tasks—fails to improve robot manipulation performance when actions are represented as discrete tokens. The authors identify a fundamental bottleneck they call the 'Compression Gap': when actions use a fixed-capacity codebook (like in OAT models), the codebook becomes the information bottleneck, preventing improvements in the vision encoder from propagating to better control. In contrast, when actions are continuous (like Diffusion Policy), scaling the vision encoder directly improves performance because the encoder itself is the constraint. This finding challenges the assumption that better vision understanding automatically translates to better robot control.
Core Technical Contribution
The paper's core contribution is the identification and formalization of the 'Compression Gap'—an information-theoretic principle explaining why scaling vision encoders has asymmetric effects depending on action representation. The authors prove that in any visuomotor pipeline, the tightest information bottleneck determines scaling behavior: discrete action codebooks with fixed capacity create a hard ceiling on performance gains from vision improvements, while continuous action representations allow those gains to propagate. This reframes the scaling problem from 'how much better is the vision encoder?' to 'where is the binding constraint in the pipeline?' The insight suggests that practitioners should co-design vision encoders and action representations jointly rather than treating them independently.
How It Works
The Compression Gap operates as a principle that identifies information bottlenecks in visuomotor systems through the lens of information theory. When actions are discretized into a fixed-capacity codebook (e.g., k-means clustering of action space into m tokens), the codebook acts as a lossy compression layer that irreversibly discards action information—this becomes the narrowest point in the pipeline regardless of vision encoder quality. Conversely, in continuous action representations like Diffusion Policy, the vision encoder extracts features that directly condition diffusion steps, making the vision features themselves the bottleneck. The paper likely analyzes information flow by measuring mutual information I(vision features; actions) and comparing bottleneck capacity under different architectures. The mechanism explains that upgrading vision encoders only helps when downstream components (action representation, decoders) have sufficient capacity to utilize that richer information.
Production Impact
For robotics teams building manipulation systems, this paper directly impacts architecture decisions: if your system uses discrete action tokenization (common in transformer-based policies), investing heavily in vision encoder scaling will yield diminishing returns—you should instead increase codebook capacity or switch to continuous representations. This changes resource allocation significantly: rather than using the largest available vision backbone (ViT-G, CLIP, etc.), you should match vision encoder complexity to your action representation capacity. Production systems would benefit from conducting information-theoretic audits on their pipelines to identify the actual constraint, then co-optimizing that component. The practical trade-off is that continuous representations like Diffusion Policy may increase inference latency and compute compared to discrete tokenization, but will actually benefit from better vision encoders, potentially justifying the investment. This also suggests that multi-scale action representations (hierarchical codebooks or hybrid discrete-continuous schemes) might unlock better scaling.
Limitations and When Not to Use This
The paper's analysis applies primarily to end-to-end visuomotor learning and may not generalize to hierarchical control systems where different components (perception, planning, control) are trained separately. It assumes fixed-capacity codebooks are the bottleneck, but doesn't fully address how learned or adaptive codebooks (VQ-VAE variants) or nested hierarchical action spaces might change the picture. The information-theoretic framework is presented conceptually but the paper may lack empirical validation across a comprehensive range of vision encoders and action representations—real-world scaling curves might not perfectly match the theoretical predictions due to optimization dynamics. Additionally, the analysis doesn't address how data efficiency interacts with the Compression Gap: a smaller vision encoder might be easier to train with limited robot data, potentially offsetting the theoretical performance ceiling from discrete actions in practice.
Research Context
This work builds on the rapid scaling success in vision-language models (CLIP, ALIGN) and extends that intuition to embodied AI, specifically vision-language-action models that have shown promise for robotic learning. It relates to concurrent research on action representations in imitation learning and reinforcement learning, where the choice between discrete tokens (Gato, OAT) and continuous outputs (Diffusion Policy, BC-Z) has been largely empirical. The paper contextualizes within information bottleneck theory (Tishby et al.) and applies it to the robotics domain, opening a new lens for analyzing visuomotor scaling laws analogous to scaling laws in language modeling. This direction likely influences future work on co-designing vision and action components, and informs the debate around whether robotics should adopt transformer-discrete-action architectures or diffusion-based continuous models.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::