BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidence
| Authors | Sean Wu et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2604.03216 |
| Download | |
| Categories | cs.CL |
Abstract
Large language models (LLMs) often produce confident but incorrect answers in settings where abstention would be safer. Standard evaluation protocols, however, require a response and do not account for how confidence should guide decisions under different risk preferences. To address this gap, we introduce the Behavioral Alignment Score (BAS), a decision-theoretic metric for evaluating how well LLM confidence supports abstention-aware decision making. BAS is derived from an explicit answer-or-abstain utility model and aggregates realized utility across a continuum of risk thresholds, yielding a measure of decision-level reliability that depends on both the magnitude and ordering of confidence. We show theoretically that truthful confidence estimates uniquely maximize expected BAS utility, linking calibration to decision-optimal behavior. BAS is related to proper scoring rules such as log loss, but differs structurally: log loss penalizes underconfidence and overconfidence symmetrically, whereas BAS imposes an asymmetric penalty that strongly prioritizes avoiding overconfident errors. Using BAS alongside widely used metrics such as ECE and AURC, we then construct a benchmark of self-reported confidence reliability across multiple LLMs and tasks. Our results reveal substantial variation in decision-useful confidence, and while larger and more accurate models tend to achieve higher BAS, even frontier models remain prone to severe overconfidence. Importantly, models with similar ECE or AURC can exhibit very different BAS due to highly overconfident errors, highlighting limitations of standard metrics. We further show that simple interventions, such as top- confidence elicitation and post-hoc calibration, can meaningfully improve confidence reliability. Overall, our work provides both a principled metric and a comprehensive benchmark for evaluating LLM confidence reliability.
Engineering Breakdown
Plain English
This paper addresses a critical gap in how we evaluate large language models: most benchmarks force models to always answer, even when uncertainty is high, but in real applications you often want the model to abstain when unsure. The authors introduce Behavioral Alignment Score (BAS), a metric that measures how well a model's confidence calibration supports better decision-making across different risk tolerances. Rather than a single score, BAS aggregates utility across a range of confidence thresholds, capturing how reliable the model's uncertainty estimates are for actual deployment decisions. They prove theoretically that perfectly calibrated confidence is the unique way to maximize BAS utility, grounding the metric in sound decision theory.
Core Technical Contribution
The core novelty is formalizing abstention-aware evaluation through decision-theoretic principles. Instead of treating confidence as a binary threshold (answer or abstain), BAS models it as a continuous utility function that depends on both the magnitude and relative ordering of confidence scores across different decision thresholds. This is fundamentally different from standard metrics like accuracy or F1 score, which ignore abstention entirely, and from existing calibration metrics like expected calibration error, which don't account for downstream decision consequences. The key insight is that confidence should be evaluated based on how it enables better decisions across varying cost-benefit trade-offs, not just whether it predicts correctness.
How It Works
BAS starts with an explicit utility model where answering correctly yields +1 utility, answering incorrectly yields -1, and abstaining yields 0. For each instance, the model produces a confidence score; instances with confidence above a decision threshold are answered, others are abstained from. The metric then sweeps across a continuum of possible thresholds and computes the realized utility at each threshold (fraction correct above threshold times +1, plus fraction incorrect above threshold times -1). BAS aggregates these utilities across all thresholds into a single score, weighted by a risk preference distribution. The authors prove that this aggregation is maximized when the model's confidence directly reflects its true probability of correctness, incentivizing calibrated uncertainty rather than overconfident predictions.
Production Impact
This changes how you'd evaluate models before deployment in risk-sensitive applications like medical diagnosis, financial decisions, or content moderation, where wrong answers sometimes cost more than admitting uncertainty. Rather than optimizing purely for accuracy, you'd optimize for confidence calibration under BAS, which directly maps to downstream business value—you can tune your abstention threshold based on actual cost-benefit constraints. In production, you'd measure BAS across different user populations or risk profiles to understand reliability under varying tolerance levels, replacing binary accept/reject decisions with confidence-driven routing. The trade-off is computational: you need to sweep many thresholds to compute BAS accurately, though you only do this at evaluation time, not inference. Integration requires confidence scores from your model output pipeline and clear specification of your utility model (what a mistake costs vs. what abstaining costs).
Limitations and When Not to Use This
The paper's theoretical result assumes perfectly specified utility models and true probability calibration, which are unrealistic; real systems have uncertain costs and models don't perfectly match their stated confidences. BAS doesn't directly address how to train models to optimize for calibration at scale—the paper establishes what to measure but not necessarily how to achieve it efficiently during training. The metric also assumes confidence scores are well-defined and comparable across different answer types, which breaks down in open-ended generation where you might not have a clear notion of confidence per token or output. The approach requires a predefined range of risk thresholds and utility weights, meaning you must know your decision-theoretic preferences upfront rather than discovering them post-hoc from data.
Research Context
This work builds on a long tradition of calibration research in uncertainty quantification and extends recent work on confidence in LLMs (particularly following papers on hallucination detection and selective prediction). It connects to decision theory literature on cost-sensitive learning and to practical research on abstention mechanisms in NLP. The paper likely benchmarks against existing uncertainty quantification methods and calibration metrics on standard datasets, demonstrating that BAS better predicts downstream task performance under abstention. This opens a research direction toward training procedures explicitly optimized for confidence calibration rather than accuracy alone, and toward richer decision-theoretic evaluation frameworks for LLMs in high-stakes domains.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::