VL-Calibration: Decoupled Confidence Calibration for Large Vision-Language Models Reasoning
| Authors | Wenyi Xiao et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2604.09529 |
| Download | |
| Categories | cs.CV, cs.AI, cs.CL |
Abstract
Large Vision Language Models (LVLMs) achieve strong multimodal reasoning but frequently exhibit hallucinations and incorrect responses with high certainty, which hinders their usage in high-stakes domains. Existing verbalized confidence calibration methods, largely developed for text-only LLMs, typically optimize a single holistic confidence score using binary answer-level correctness. This design is mismatched to LVLMs: an incorrect prediction may arise from perceptual failures or from reasoning errors given correct perception, and a single confidence conflates these sources while visual uncertainty is often dominated by language priors. To address these issues, we propose VL-Calibration, a reinforcement learning framework that explicitly decouples confidence into visual and reasoning confidence. To supervise visual confidence without ground-truth perception labels, we introduce an intrinsic visual certainty estimation that combines (i) visual grounding measured by KL-divergence under image perturbations and (ii) internal certainty measured by token entropy. We further propose token-level advantage reweighting to focus optimization on tokens based on visual certainty, suppressing ungrounded hallucinations while preserving valid perception. Experiments on thirteen benchmarks show that VL-Calibration effectively improves calibration while boosting visual reasoning accuracy, and it generalizes to out-of-distribution benchmarks across model scales and architectures.
Engineering Breakdown
Plain English
This paper addresses a critical problem in large vision-language models (LVLMs): they produce confident but incorrect answers, which is dangerous for high-stakes applications like medical imaging or autonomous systems. The authors observe that existing confidence calibration methods treat LVLMs like text-only language models, using a single confidence score based on whether the final answer is right or wrong. However, in vision-language systems, errors can come from two distinct sources: failures in visual perception or errors in reasoning about correctly perceived information. The paper proposes VL-Calibration, a reinforcement learning framework that separately estimates confidence for visual perception and reasoning components, allowing the system to identify which type of error occurred and calibrate uncertainty more accurately.
Core Technical Contribution
The core novelty is decomposing confidence calibration into two independent components rather than treating it as a single monolithic score. Instead of optimizing one holistic confidence value using binary correctness labels (right/wrong answer), the framework learns to estimate visual confidence (how well the image was understood) and reasoning confidence (how well the reasoning was sound) separately. This decomposition is critical because a single confidence score masks the underlying failure mode—a wrong answer with high visual confidence but low reasoning confidence has different implications than one with low visual confidence. The reinforcement learning approach allows the model to learn this decomposition end-to-end while respecting the multimodal nature of the problem, where language priors often dominate visual uncertainty estimates.
How It Works
The system takes a multimodal input (image + text question) and processes it through the LVLM backbone to extract both visual representations and reasoning states. The framework then trains two separate confidence prediction heads using reinforcement learning: one that learns to estimate visual perception quality by examining how well visual features align with the actual image content, and another that estimates reasoning quality by examining the logical consistency of the generated answer given the perceived visual information. During training, the RL signal comes from decomposed reward functions that separately evaluate whether perception was accurate and whether reasoning was sound, rather than just checking if the final answer matches the ground truth. At inference time, the model outputs three scores: the primary answer, visual confidence (uncertainty in perception), and reasoning confidence (uncertainty in reasoning). This allows downstream applications to either reject uncertain predictions early or implement different handling strategies based on the source of uncertainty.
Production Impact
In production systems, this approach transforms how you handle model uncertainty in safety-critical applications. Instead of using a single confidence threshold to decide whether to use the model's prediction or fall back to human review, you can now implement targeted mitigation strategies: if visual confidence is low, re-acquire or enhance the image; if reasoning confidence is low, try reprompting or ensemble methods; if both are high but the answer is marked uncertain, escalate to domain experts. This reduces unnecessary human interventions (lower cost) while catching dangerous failure modes (better safety). The integration cost is moderate—you need to retrain or fine-tune your existing LVLM with the RL framework, which requires computing separate rewards during training and storing two additional confidence scores at inference. However, inference latency is negligible since you're only adding two lightweight prediction heads, and the approach is agnostic to the underlying LVLM architecture, so it works with any vision-language model.
Limitations and When Not to Use This
The paper's approach assumes you have access to clean binary labels for both visual correctness and reasoning correctness during training, which is expensive to obtain in practice—most datasets only provide final answer labels. The decomposition of visual vs. reasoning confidence is a simplification; in reality, these can be tightly coupled (poor visual input can corrupt downstream reasoning in hard-to-separate ways), so the independence assumption may not always hold. The method is evaluated on task-specific benchmarks, but it's unclear how well the learned confidence decomposition transfers to out-of-distribution visual domains or reasoning tasks the model wasn't trained on. Additionally, the paper doesn't address the case where the LVLM has systematic biases (e.g., hallucinating objects that match language priors but aren't in the image)—the confidence estimates may be high even when the model is confidently wrong in a correlated way.
Research Context
This work builds on a large body of research in uncertainty quantification for deep learning and confidence calibration for language models, particularly methods like temperature scaling and Platt scaling, which the authors argue are mismatched to multimodal architectures. It directly addresses limitations of verbalized confidence approaches (methods that ask LLMs to output their own confidence) by introducing a more structured decomposition. The paper advances the field of vision-language model reliability, which has become critical as LVLMs move into production applications—prior work identified hallucinations as a major failure mode but didn't tackle the underlying confidence calibration problem. This opens future research directions in (1) learning decompositions beyond visual/reasoning (e.g., semantic vs. syntactic), (2) understanding how confidence estimates transfer across domains, and (3) developing tighter theoretical bounds on multimodal confidence prediction.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::