Hypothesis Testing over Observable Regimes in Singular Models
| Authors | Sean Plummer |
| Year | 2026 |
| Field | Statistics / ML |
| arXiv | 2602.24165 |
| Download | |
| Categories | stat.ML |
Abstract
Hypothesis testing in singular statistical models is often regarded as inherently problematic due to non-identifiability and degeneracy of the Fisher information. We show that the fundamental obstruction to testing in such models is not singularity itself, but the formulation of hypotheses on non-identifiable parameter quantities. Testing is inherently a problem in distribution space: if two hypotheses induce overlapping subsets of the model class, then no uniformly consistent test exists. We formalize this overlap obstruction and show that hypotheses depending on non-identifiable parameter functions necessarily fail in this sense. In contrast, hypotheses formulated over identifiable observables-quantities that are determined by the induced distribution-reduce entirely to classical testing theory. When the corresponding distributional regimes are separated in Hellinger distance, uniformly consistent tests exist and posterior contraction follows from standard testing-based arguments. Near singular boundaries, separation may collapse locally, leading to scale-dependent detectability governed jointly by sample size and distance to the singular stratum. We illustrate these phenomena in Gaussian mixture models and reduced-rank regression, exhibiting both untestable non-identifiable hypotheses and classically testable identifiable ones. The results provide a structural classification of which hypotheses in singular models are statistically meaningful.
Engineering Breakdown
Plain English
This paper solves a fundamental problem in statistical testing: hypothesis testing on singular models—statistical models with non-identifiable parameters and degenerate Fisher information matrices. The authors show that the real obstruction to reliable testing isn't singularity itself, but rather formulating hypotheses on non-identifiable parameter quantities. They prove that if two competing hypotheses induce overlapping subsets of probability distributions in the model class, no uniformly consistent test can distinguish between them. The key finding is that testing works perfectly well when hypotheses are formulated over observable, identifiable quantities—those determined entirely by the induced distribution—reducing the problem to classical statistical testing with standard guarantees.
Core Technical Contribution
The paper's core insight is a fundamental distinction between testability and model singularity: singular models are not inherently untestable, but hypotheses on non-identifiable parameters are. The authors formalize this through an 'overlap obstruction'—when two hypotheses generate overlapping sets of probability distributions, no test can uniformly distinguish them regardless of sample size. The technical novelty is proving that this obstruction is necessary and sufficient for test failure, and conversely, that hypotheses formulated entirely on identifiable observables (distribution-level quantities) are always testable using classical techniques. This reframes the problem from a model geometry issue to a hypothesis formulation issue, providing constructive guidance for practitioners.
How It Works
The paper operates in distribution space rather than parameter space. Given a singular statistical model with parameter vector θ, the mechanism works as follows: (1) For any hypothesis pair (H₀, H₁), determine what subsets of probability distributions each induces—call these D₀ and D₁. (2) Check whether D₀ and D₁ overlap: if P ∈ D₀ ∩ D₁, then no test can achieve uniform consistency because the same observed data could have generated either hypothesis true. (3) Formalize a hypothesis as depending on an observable function φ(θ) if φ's value is uniquely determined by the induced distribution P(·;θ), independent of unidentifiable parameters. (4) Show that when both H₀ and H₁ depend entirely on identifiable observables, their induced distribution sets D₀ and D₁ are separable, and standard classical testing applies. The key insight is that identifiability at the distribution level, not parameter level, determines testability.
Production Impact
For engineers building ML systems that rely on hypothesis testing (A/B testing, model selection, anomaly detection), this provides a diagnostic tool: before investing in complex testing infrastructure, determine whether your hypothesis is formulated on identifiable quantities. In production, this means: (1) when testing hypotheses on latent representations or unobservable model parameters, no amount of data will give uniformly reliable test results—you must reformulate your hypothesis in terms of observable predictions or outcomes; (2) conversely, when hypotheses target observable quantities (prediction accuracy, distribution moments, likelihood ratios on observable data), you can use standard frequentist or Bayesian testing with theoretical guarantees; (3) this avoids wasted engineering effort building tests for inherently non-testable hypotheses. The trade-off is the upfront analytical work to verify a hypothesis depends only on identifiable observables, but this prevents costly downstream testing failures.
Limitations and When Not to Use This
The paper assumes you can cleanly separate identifiable from non-identifiable components of a hypothesis, which may be difficult in high-dimensional or complex models where identifiability structure is unknown. It doesn't provide algorithms for automatically detecting whether a proposed hypothesis is testable—that requires manual analysis of the model and hypothesis. The theory is purely asymptotic (uniform consistency as n→∞) and doesn't address finite-sample testing power or practical sample complexity. The paper also doesn't handle composite observables that mix identifiable and non-identifiable components, leaving open the question of partial testability. Furthermore, in modern deep learning systems with learned representations, the notion of 'observable' is ambiguous—is a learned feature observable or latent?—which limits direct applicability.
Research Context
This work builds on classical results in singular model theory and information geometry, particularly Watanabe's singular learning theory, which studies the asymptotic behavior of models where the Fisher information matrix is degenerate. The paper extends and clarifies understanding of why some singular models (like Bayesian neural networks, mixture models, and hidden Markov models) still admit valid inference despite singularity. It relates to identifiability theory in causal inference and latent variable models, which has long grappled with testing challenges. The contribution opens a research direction toward constructive methods for reformulating non-testable hypotheses into testable observables, and potentially toward automatic detectability of hypothesis identifiability in complex learned models.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::