LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Models
| Authors | Chanyoung Kim et al. |
| Year | 2026 |
| HF Upvotes | 81 |
| arXiv | 2603.28301 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Vision-Language-Action (VLA) models achieve strong performance in robotic manipulation by leveraging pre-trained vision-language backbones. However, in downstream robotic settings, they are typically fine-tuned with limited data, leading to overfitting to specific instruction formulations and leaving robustness to paraphrased instructions underexplored. To study this gap, we introduce LIBERO-Para, a controlled benchmark that independently varies action expressions and object references for fine-grained analysis of linguistic generalization. Across seven VLA configurations (0.6B-7.5B), we observe consistent performance degradation of 22-52 pp under paraphrasing. This degradation is primarily driven by object-level lexical variation: even simple synonym substitutions cause large drops, indicating reliance on surface-level matching rather than semantic grounding. Moreover, 80-96% of failures arise from planning-level trajectory divergence rather than execution errors, showing that paraphrasing disrupts task identification. Binary success rate treats all paraphrases equally, obscuring whether models perform consistently across difficulty levels or rely on easier cases. To address this, we propose PRIDE, a metric that quantifies paraphrase difficulty using semantic and syntactic factors. Our benchmark and corresponding code are available at: https://github.com/cau-hai-lab/LIBERO-Para
Engineering Breakdown
Plain English
This paper identifies a critical robustness gap in Vision-Language-Action (VLA) models used for robot control. While these models perform well when fine-tuned on limited robotic datasets, they degrade significantly when instructions are paraphrased—even simple synonym substitutions cause performance drops of 22-52 percentage points across models ranging from 0.6B to 7.5B parameters. The authors introduce LIBERO-Para, a controlled benchmark that systematically varies action expressions and object references independently, enabling fine-grained analysis of where linguistic generalization breaks down. They discover that object-level lexical variation is the primary culprit, meaning the model's understanding of what object to manipulate is brittle despite its strong vision-language pretraining.
Core Technical Contribution
The core contribution is LIBERO-Para, a controlled benchmark specifically designed to isolate and measure linguistic generalization failures in VLA models through independent manipulation of action expressions and object references. Unlike existing robotic benchmarks that bundle multiple sources of variation together, LIBERO-Para enables researchers to pinpoint exactly which linguistic phenomena cause degradation—revealing that object reference variation is far more damaging than action expression variation. The paper provides the first systematic study of paraphrasing robustness in vision-language-action models, filling a gap between the known robustness of vision-language models on images and the actual robustness of fine-tuned VLAs on embodied tasks. This diagnostic approach is novel because it treats linguistic generalization as a measurable, decoupled problem rather than an implicit byproduct of general VLA training.
How It Works
LIBERO-Para operates by taking existing robotic manipulation demonstrations and systematically rewriting instruction text while keeping the underlying action and object targets fixed. The benchmark independently varies two dimensions: (1) action expressions—different ways of describing the same manipulation (e.g., 'pick up', 'grab', 'lift'), and (2) object references—synonyms and paraphrases for the same object (e.g., 'cube' vs. 'block' vs. 'red object'). For each demonstration, researchers generate multiple paraphrased versions, creating a matrix of linguistic variants tied to identical visual scenes and action trajectories. When VLA models are fine-tuned on original instruction text and then evaluated on paraphrased versions, the benchmark measures performance degradation in isolation. The key insight is architectural: the vision and action components stay constant, so any performance drop is directly attributable to the language encoder's failure to recognize semantic equivalence, not to failures in visual grounding or action prediction.
Production Impact
For engineers deploying VLAs in production robot systems, this paper signals a critical fragility: robots fine-tuned on small datasets will fail when operators or downstream systems use different terminology than the training data. In practice, this means a robot trained on 'place the cube in the bin' might fail on 'put the block in the container' despite identical visual and physical scenarios—a failure mode that manifests in the field, not in the lab. To mitigate this, production teams should either (1) collect training data with diverse paraphrases for each task, significantly increasing data collection overhead, (2) add a semantic normalization layer that maps synonyms before feeding to the VLA, adding 10-50ms latency, or (3) use instruction augmentation during fine-tuning, which requires careful validation to avoid introducing spurious correlations. The benchmark itself is lightweight—it requires no architectural changes or retraining, just re-evaluation on paraphrased instruction sets—making it a practical diagnostic tool for robustness testing before deployment.
Limitations and When Not to Use This
The paper does not address how to systematically fix paraphrasing brittleness, only how to measure it; proposed solutions (data augmentation, semantic normalization) are left to future work. LIBERO-Para assumes paraphrases are natural and human-like, but may not capture adversarial or out-of-distribution language (e.g., technical jargon, non-English instructions, or instructions written in uncommon syntactic structures). The benchmark focuses on object reference and action expression variation but does not test other forms of linguistic robustness such as spatial preposition variation ('in' vs. 'inside' vs. 'within'), quantity changes ('a cube' vs. 'two cubes'), or relative instructions ('the larger cube'), leaving those dimensions unexplored. Additionally, the study is limited to seven VLA configurations; it's unclear whether the 22-52 pp degradation pattern holds for newer scaling laws, multimodal pretraining objectives beyond standard vision-language alignment, or embodied models trained from scratch without vision-language pretraining.
Research Context
This work builds on the recent success of scaling vision-language models (like CLIP) to embodied control through VLA architectures, but highlights a critical blind spot: pretraining robustness does not transfer to instruction robustness in downstream fine-tuning. It extends research on robotic language grounding and instruction following, which traditionally focused on visual grounding correctness rather than linguistic robustness. The paper is part of a broader conversation about robustness and generalization in multimodal models, adjacent to recent work on out-of-distribution detection in vision-language models and adversarial robustness in language understanding. LIBERO-Para positions itself as a diagnostic benchmark in the vein of other controlled evaluation suites (like ScanNet for 3D grounding or COCO for vision-language alignment), enabling the community to measure a specific, previously unmeasured form of generalization and spurring follow-up work on instruction robustness in embodied AI.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::