Faithful GRPO: Improving Visual Spatial Reasoning in Multimodal Language Models via Constrained Policy Optimization
| Authors | Sai Srinivas Kancheti et al. |
| Year | 2026 |
| HF Upvotes | 6 |
| arXiv | 2604.08476 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Multimodal reasoning models (MRMs) trained with reinforcement learning with verifiable rewards (RLVR) show improved accuracy on visual reasoning benchmarks. However, we observe that accuracy gains often come at the cost of reasoning quality: generated Chain-of-Thought (CoT) traces are frequently inconsistent with the final answer and poorly grounded in the visual evidence. We systematically study this phenomenon across seven challenging real-world spatial reasoning benchmarks and find that it affects contemporary MRMs such as ViGoRL-Spatial, TreeVGR as well as our own models trained with standard Group Relative Policy Optimization (GRPO). We characterize CoT reasoning quality along two complementary axes: "logical consistency" (does the CoT entail the final answer?) and "visual grounding" (does each reasoning step accurately describe objects, attributes, and spatial relationships in the image?). To address this, we propose Faithful GRPO (FGRPO), a variant of GRPO that enforces consistency and grounding as constraints via Lagrangian dual ascent. FGRPO incorporates batch-level consistency and grounding constraints into the advantage computation within a group, adaptively adjusting the relative importance of constraints during optimization. We evaluate FGRPO on Qwen2.5-VL-7B and 3B backbones across seven spatial datasets. Our results show that FGRPO substantially improves reasoning quality, reducing the inconsistency rate from 24.5% to 1.7% and improving visual grounding scores by +13%. It also improves final answer accuracy over simple GRPO, demonstrating that faithful reasoning enables better answers.
Engineering Breakdown
Plain English
undefined
Core Technical Contribution
The paper's central contribution is the systematic characterization and quantification of a previously under-studied failure mode in RLVR-trained multimodal models: the decoupling between answer correctness and reasoning quality. Rather than proposing a new training algorithm, the authors establish a diagnostic framework that measures both logical consistency and visual grounding as separate dimensions, revealing that standard RLVR optimization (including GRPO) inadvertently incentivizes spurious reasoning patterns. This work is novel because prior research focused on improving end-task accuracy, missing that RLVR creates a misalignment where models learn to predict correct answers without developing interpretable or reliable reasoning. The characterization across seven real-world benchmarks provides empirical evidence that this isn't a bug in any single model architecture, but a fundamental property of the RLVR training objective.
How It Works
The methodology involves training multimodal reasoning models on visual reasoning tasks using reinforcement learning with verifiable rewards, where the reward signal comes from checking whether the final answer matches ground truth on test sets. During training, models learn to generate chain-of-thought (CoT) reasoning traces followed by a final answer, optimizing for maximum reward. The authors then measure the quality of generated reasoning through two evaluation dimensions: (1) logical consistency—typically implemented by prompting a language model to check whether the stated reasoning actually entails the final answer, and (2) visual grounding—verifying that intermediate reasoning steps align with detectable objects, spatial relationships, or features visible in the actual image. The key finding emerges from comparing these two quality metrics against end-task accuracy: they observe that models can achieve high accuracy while simultaneously producing low-consistency or poorly-grounded reasoning, indicating that the RLVR objective doesn't sufficiently constrain the internal reasoning process. The evaluation is performed on established spatial reasoning benchmarks (likely including datasets like GQA, CLEVR-based variants, or similar spatially-annotated visual question answering datasets).
Production Impact
For engineers deploying multimodal reasoning systems in production, this research has immediate implications for trust and debuggability. If your system achieves 85% accuracy on a benchmark but the reasoning traces are internally inconsistent, you cannot safely deploy it in domains requiring explainability (medical imaging, autonomous systems, legal review). This means production pipelines need to instrument both answer correctness and reasoning quality checks—you cannot rely on benchmark accuracy alone as a proxy for system reliability. The work suggests that straightforward RLVR training will create models that fail silently: they produce confident-sounding justifications for correct answers that don't actually follow logically, making it extremely difficult to identify failure modes through human review. To address this in production, you would need to either (1) augment the RLVR training objective to explicitly reward consistent and grounded reasoning, requiring additional computational cost to verify reasoning quality during training, or (2) implement post-hoc reasoning validation layers that check consistency and visual grounding before returning predictions, adding latency to inference. The latter approach adds ~200-500ms per request depending on your reasoning verification model.
Limitations and When Not to Use This
The paper diagnoses the problem but does not propose a complete solution for fixing the RLVR objective itself, leaving open the question of whether you can train models that are both accurate and reason transparently. The evaluation relies on automated metrics for logical consistency and visual grounding (likely using language model checkers and vision model verifiers), which themselves may be imperfect or biased, potentially underestimating or overestimating the severity of the issue. The study is limited to spatial reasoning tasks on existing benchmarks; it remains unclear whether this failure mode extends equally to other reasoning domains (numerical, commonsense, temporal) or to models trained with other RLVR variants beyond GRPO. Additionally, the work does not address scalability questions: as models get larger and reasoning traces become more complex, the cost of verifying consistency and grounding during training could become prohibitive, and the proposed evaluation metrics may become less reliable. Finally, the paper doesn't deeply explore why RLVR specifically creates this decoupling—whether it's an inherent property of reward shaping, insufficient training signal diversity, or an artifact of model capacity deserving deeper investigation.
Research Context
This paper builds on recent progress in reinforcement learning for language model improvement (e.g., RLHF, DPO, GRPO) and extends these ideas to multimodal settings where visual reasoning requires both semantic understanding and spatial grounding. It addresses a gap identified by prior work on chain-of-thought reasoning (Wei et al., Kojima et al.) which showed CoT improves accuracy but didn't rigorously study whether the generated reasoning is actually valid. The work relates to broader safety and alignment concerns in ML: similar to adversarial robustness research, it reveals that optimizing a narrow objective (final answer correctness) can create unexpected failure modes in other important properties (reasoning validity). The paper implicitly opens a new research direction—developing training objectives that jointly optimize for both accuracy and reasoning quality—similar to how recent work in mechanistic interpretability and concept-based explanations has pushed toward more interpretable models. The use of multiple spatial reasoning benchmarks aligns with the current trend of evaluation rigor, where a single benchmark is considered insufficient to validate new methods.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::