Effective sample size approximations as entropy measures
| Authors | L. Martino & V. Elvira |
| Year | 2026 |
| Field | AI / ML |
| arXiv | 2602.22954 |
| Download | |
| Categories | cs.CE, stat.CO, stat.ML |
Abstract
In this work, we analyze alternative effective sample size (ESS) metrics for importance sampling algorithms, and discuss a possible extended range of applications. We show the relationship between the ESS expressions used in the literature and two entropy families, the Rényi and Tsallis entropy. The Rényi entropy is connected to the Huggins-Roy's ESS family introduced in \cite{Huggins15}. We prove that that all the ESS functions included in the Huggins-Roy's family fulfill all the desirable theoretical conditions. We analyzed and remark the connections with several other fields, such as the Hill numbers introduced in ecology, the Gini inequality coefficient employed in economics, and the Gini impurity index used mainly in machine learning, to name a few. Finally, by numerical simulations, we study the performance of different ESS expressions contained in the previous ESS families in terms of approximation of the theoretical ESS definition, and show the application of ESS formulas in a variable selection problem.
Engineering Breakdown
Plain English
This paper establishes a formal mathematical bridge between effective sample size (ESS) metrics used in importance sampling and two fundamental entropy families: Rényi and Tsallis entropy. The authors prove that all ESS functions in the Huggins-Roy family satisfy desirable theoretical properties and demonstrate connections to concepts from ecology (Hill numbers), economics (Gini coefficient), and machine learning (Gini impurity). The work extends the applicability of ESS metrics beyond traditional importance sampling by showing they can be understood through the lens of entropy, enabling practitioners to choose ESS metrics more systematically based on problem-specific entropy properties.
Core Technical Contribution
The core novelty is the theoretical unification of disparate ESS metrics under two entropy frameworks (Rényi and Tsallis), proving that the Huggins-Roy ESS family maintains all desired mathematical properties. This connection was not previously established in the literature and provides a principled way to derive and validate new ESS variants. The paper also reveals unexpected cross-domain relationships—showing that concepts from ecology, economics, and machine learning are mathematically equivalent to different parameterizations of entropy. This enables practitioners to leverage domain-specific insights from these fields when designing sampling algorithms.
How It Works
The paper starts by analyzing existing ESS definitions used in importance sampling, which measure how many effective independent samples an importance sampling algorithm produces from a weighted sample set. The authors then reformulate these ESS expressions mathematically and show they can be expressed as functions of Rényi entropy (a one-parameter family of entropy measures) and Tsallis entropy (another parameterized entropy family). For each ESS metric, they identify which entropy order (α parameter) it corresponds to, creating a mapping between ESS literature and entropy theory. They then verify that all ESS functions in the Huggins-Roy family satisfy desirable properties: they're bounded between 1 and N (sample size), equal 1 when weights are maximally unequal, and equal N when weights are uniform. The analysis also draws parallels to Hill numbers (used in ecology to measure species diversity), the Gini coefficient (measuring income inequality), and Gini impurity (used in decision trees), showing these are all instances of the same mathematical family with different interpretations.
Production Impact
For engineers building particle filters, variational inference, or any system using importance sampling, this work provides a principled framework for selecting and designing ESS metrics tailored to specific problems. Instead of ad-hoc ESS choice, practitioners can now select an entropy order (α) that matches their application's requirements—lower α values may be appropriate when you care about the effective number of unique samples, while higher values work better when you want robustness to extreme weights. The unification enables reuse of theoretical results: if you prove a property for one Rényi entropy order, you gain insights across the entire family, reducing redundant analysis. The cross-domain connections (ecology, economics) open possibilities for adapting diversity/inequality metrics from other fields into sampling algorithms. The main trade-off is increased implementation complexity—practitioners must understand entropy families and parameter selection—but the computational cost of ESS computation itself remains negligible compared to sampling operations.
Limitations and When Not to Use This
The paper does not address how to automatically select the optimal entropy order (α parameter) for a given problem—practitioners still face a hyperparameter selection challenge. The theoretical results assume discrete probability distributions and may not directly extend to continuous sampling scenarios without additional work. The practical guidance on when to use Rényi versus Tsallis versus other entropy families is limited; the paper maps the mathematics but doesn't provide clear decision rules for practitioners. Additionally, the connections to Hill numbers and Gini coefficients, while intellectually interesting, lack empirical validation showing that adapting ecological or economic insights actually improves importance sampling performance in realistic machine learning applications.
Research Context
This work builds directly on the Huggins-Roy ESS framework from prior research and extends it by connecting to the broader literature on entropy families, which have been studied extensively in information theory. The paper sits at the intersection of importance sampling (a foundational Monte Carlo technique) and information-theoretic measures, connecting previously separate research communities. It opens research directions toward entropy-guided algorithm design, where sampling algorithms could adaptively select or interpolate between different entropy orders during execution. The unification also suggests that theoretical results about entropy (e.g., asymptotic properties, convergence rates) may directly apply to ESS-based algorithms, potentially accelerating future theoretical analysis.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::