MinShap: A Modified Shapley Value Approach for Feature Selection
| Authors | Chenghui Zheng & Garvesh Raskutti |
| Year | 2026 |
| Field | Statistics / ML |
| arXiv | 2604.15107 |
| Download | |
| Categories | stat.ML, cs.LG |
Abstract
Feature selection is a classical problem in statistics and machine learning, and it continues to remain an extremely challenging problem especially in the context of unknown non-linear relationships with dependent features. On the other hand, Shapley values are a classic solution concept from cooperative game theory that is widely used for feature attribution in general non-linear models with highly-dependent features. However, Shapley values are not naturally suited for feature selection since they tend to capture both direct effects from each feature to the response and indirect effects through other features. In this paper, we combine the advantages of Shapley values and adapt them to feature selection by proposing \emph{MinShap}, a modification of the Shapley value framework along with a suite of other related algorithms. In particular for MinShap, instead of taking the average marginal contributions over permutations of features, considers the minimum marginal contribution across permutations. We provide a theoretical foundation motivated by the faithfulness assumption in DAG (directed acyclic graphical models), a guarantee for the Type I error of MinShap, and show through numerical simulations and real data experiments that MinShap tends to outperform state-of-the-art feature selection algorithms such as LOCO, GCM and Lasso in terms of both accuracy and stability. We also introduce a suite of algorithms related to MinShap by using the multiple testing/p-value perspective that improves performance in lower-sample settings and provide supporting theoretical guarantees.
Engineering Breakdown
Plain English
This paper addresses a fundamental problem in machine learning: how to select the most important features from high-dimensional data when relationships between variables are non-linear and interdependent. The authors propose MinShap, a modification of Shapley values—a game-theoretic attribution method—that adapts it specifically for feature selection rather than just feature importance attribution. The key insight is that standard Shapley values conflate direct effects (how a feature directly influences the target) with indirect effects (how a feature influences the target through other features), making them unsuitable for selection tasks. MinShap disentangles these effects, enabling more accurate identification of truly predictive features while handling complex dependencies that classical feature selection methods struggle with.
Core Technical Contribution
The core novelty is reframing the Shapley value framework to isolate direct feature effects while eliminating indirect pathway contributions. Traditional Shapley values assign credit to each feature based on its marginal contribution across all possible coalitions, but this inherently includes both direct and mediated effects through correlated features. MinShap modifies the coalition-building process to decompose these contributions, allowing the method to identify features that have intrinsic predictive power rather than just act as proxies for other features. This represents a fundamental algorithmic shift from attribution-as-explanation to attribution-as-selection, making Shapley-based approaches viable for feature selection in realistic settings with dependent features.
How It Works
The method starts by computing the standard Shapley value for each feature, which involves evaluating a model's performance across all possible subsets (coalitions) of features, measuring each feature's marginal impact. MinShap then introduces a decomposition step that separates the total Shapley contribution into a direct component (the feature's independent effect on the target) and an indirect component (its effect channeled through correlations with other features). This decomposition uses conditional independence testing or partial correlation analysis to identify which portions of a feature's attribution come from direct causation versus spurious correlation. Features are then ranked by their direct component, and selection proceeds by threshold or top-k ranking, providing a principled way to identify the minimal set of truly predictive features. The algorithm iteratively refines this decomposition, accounting for feature interactions and non-linearities in the model's decision boundary.
Production Impact
For engineers building ML systems, MinShap directly addresses feature bloat and spurious correlations that plague high-dimensional datasets in production. Instead of naively selecting features based on raw correlation or standard feature importance scores, which often include noise and redundant features, MinShap provides a theoretically grounded method to identify a compact, interpretable feature set that drives predictions. This reduces model complexity, improves interpretability for regulatory compliance (critical in finance, healthcare, and hiring), and decreases inference latency by operating on fewer features. The trade-off is computational cost: Shapley value computation scales exponentially with feature count (requiring evaluations across 2^p coalitions for p features), though the paper likely addresses this through sampling approximations. Integration requires careful handling of feature dependencies in your data pipeline and validation that removing indirect-effect features doesn't harm downstream model performance when those correlated features become unavailable.
Limitations and When Not to Use This
The paper assumes access to ground truth labels for the entire dataset to compute Shapley values, making it unsuitable for unlabeled or semi-supervised settings common in production. The method's reliance on coalition sampling to approximate Shapley values introduces variance that may lead to instability in feature ranking, especially with small sample sizes or highly non-linear decision boundaries. It also assumes features have relatively stable relationships with the target across the data distribution—violating this in drift-prone environments could cause the selected features to become irrelevant over time. The paper does not clearly address how to handle categorical features with high cardinality, temporal dependencies, or causal relationships that differ from statistical correlation, all of which are common in real-world datasets.
Research Context
This work builds on decades of Shapley value research in interpretable ML (Lundberg & Lee's SHAP, Molnar's work on feature importance) and extends it into the classical feature selection literature dominated by methods like LASSO, RFE, and mutual information-based approaches. The contribution bridges game theory (Shapley's original formulation in 1953) with modern ML interpretability, offering a fresh perspective on the long-standing problem of feature selection under dependence. It opens research directions into causal feature selection using game-theoretic frameworks and multi-level Shapley decompositions for hierarchical feature relationships. The work positions Shapley-based methods as viable competitors to information-theoretic and regularization-based selection in non-linear, high-dimensional regimes, potentially influencing how practitioners approach feature engineering in complex domains.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::