A note on the area under the likelihood and the fake evidence for model selection
| Authors | L. Martino & F. Llorente |
| Year | 2026 |
| Field | AI / ML |
| arXiv | 2602.22965 |
| Download | |
| Categories | stat.ME, cs.CE, stat.CO, stat.ML |
Abstract
Improper priors are not allowed for the computation of the Bayesian evidence Z=p({\bf y}) (a.k.a., marginal likelihood), since in this case is not completely specified due to an arbitrary constant involved in the computation. However, in this work, we remark that they can be employed in a specific type of model selection problem: when we have several (possibly infinite) models belonging to the same parametric family (i.e., for tuning parameters of a parametric model). However, the quantities involved in this type of selection cannot be considered as Bayesian evidences: we suggest to use the name fake evidences'' (or areas under the likelihood'' in the case of uniform improper priors). We also show that, in this model selection scenario, using a diffuse prior and increasing its scale parameter asymptotically to infinity, we cannot recover the value of the area under the likelihood, obtained with a uniform improper prior. We first discuss it from a general point of view. Then we provide, as an applicative example, all the details for Bayesian regression models with nonlinear bases, considering two cases: the use of a uniform improper prior and the use of a Gaussian prior, respectively. A numerical experiment is also provided confirming and checking all the previous statements.
Engineering Breakdown
Plain English
This paper addresses a fundamental problem in Bayesian model selection: when you have multiple models from the same parametric family (e.g., different tuning parameters of a single model type), traditional Bayesian evidence cannot be computed using improper priors because the evidence lacks a well-defined constant. The authors propose that in this specific scenario, practitioners can use improper priors to compute what they call 'fake evidences' or 'areas under the likelihood'—quantities that behave like evidence for model comparison but aren't technically Bayesian evidence. This opens a practical workaround for hyperparameter tuning and model selection within a single parametric family where classical evidence computation would normally fail.
Core Technical Contribution
The key insight is distinguishing between two different model selection problems: selecting among entirely different model families (where Bayesian evidence applies) versus selecting within a single parametric family by tuning parameters (where improper priors become viable). The authors formalize that 'fake evidence'—defined as the area under the likelihood when using uniform improper priors—can be legitimately used for within-family model selection because the arbitrary constant in improper priors cancels out during comparison. This reframes what has been considered a mathematical violation (using improper priors for evidence) into a valid technique for a specific, practically important class of problems. The contribution is primarily theoretical clarification rather than a new algorithm, but it justifies existing computational shortcuts.
How It Works
In standard Bayesian model selection, evidence Z = p(y) is computed by integrating the likelihood weighted by the prior: Z = ∫ p(y|θ)p(θ)dθ. Improper priors (like uniform priors over infinite support) introduce an undefined normalization constant, making Z ill-defined across different models. However, when selecting among models in the same parametric family (only differing in tuning parameters like regularization strength), the authors show that the ratio of these 'fake evidences' remains well-defined because the problematic constant appears identically in numerator and denominator and cancels. The 'area under the likelihood' specifically means computing ∫ p(y|θ)dθ without a proper prior, which is mathematically equivalent to using a uniform improper prior. This allows practitioners to compare models within a family using likelihood integrals alone, bypassing the need for proper prior specification.
Production Impact
For practitioners building hyperparameter tuning pipelines, this work validates using likelihood-based model selection criteria (like cross-validation approximations or marginal likelihood estimates) without carefully specifying informative priors for each candidate model. In production systems, this means you can compare thousands of parameter configurations in a parametric family using fast likelihood computations rather than waiting for expensive Bayesian posterior inference with proper priors. The practical impact is speed: likelihood integration is often cheaper than posterior sampling, making it feasible to do finer-grained hyperparameter sweeps. However, the trade-off is that this approach only works within a single parametric family—you cannot use it to compare fundamentally different model architectures (e.g., linear regression vs. neural network), limiting its scope to within-family tuning rather than model class selection.
Limitations and When Not to Use This
The paper's scope is intentionally narrow: 'fake evidence' only applies to model selection within a single parametric family, not across different model classes. The authors do not provide empirical validation showing that fake evidence consistently selects better-performing models compared to alternatives like cross-validation or information criteria in real datasets. The work also assumes that diffuse/uniform improper priors are appropriate for the problem at hand—in domains where prior information strongly constrains parameters, this assumption may not hold. Additionally, the paper does not address computational efficiency of actually computing the likelihood integrals, which can be expensive for high-dimensional parameter spaces, nor does it provide guidance on when fake evidence should be preferred over practical alternatives already used in production (cross-validation, AIC, BIC).
Research Context
This work sits within the broader literature on Bayesian model selection and the long-standing debate over the use of improper priors in Bayesian inference. It builds on classical results in Bayesian evidence computation and extends understanding of when improper priors, normally forbidden, become mathematically valid. The paper contributes to a practical gap between Bayesian theory (which is rigid about prior specification) and engineering practice (which often uses approximations and shortcuts). The research direction opened is clarifying when theoretical rules can be relaxed in specific model selection scenarios, potentially bridging the divide between strict Bayesian methods and pragmatic machine learning workflows. This is relevant to hyperparameter optimization literature, which has traditionally used non-Bayesian methods but could potentially benefit from principled Bayesian justification.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::