LoBoost: Fast Model-Native Local Conformal Prediction for Gradient-Boosted Trees
| Authors | Vagner Santos et al. |
| Year | 2026 |
| Field | Statistics / ML |
| arXiv | 2602.22432 |
| Download | |
| Categories | stat.ML, cs.LG |
Abstract
Gradient-boosted decision trees are among the strongest off-the-shelf predictors for tabular regression, but point predictions alone do not quantify uncertainty. Conformal prediction provides distribution-free marginal coverage, yet split conformal uses a single global residual quantile and can be poorly adaptive under heteroscedasticity. Methods that improve adaptivity typically fit auxiliary nuisance models or introduce additional data splits/partitions to learn the conformal score, increasing cost and reducing data efficiency. We propose LoBoost, a model-native local conformal method that reuses the fitted ensemble's leaf structure to define multiscale calibration groups. Each input is encoded by its sequence of visited leaves; at resolution level k, we group points by matching prefixes of leaf indices across the first k trees and calibrate residual quantiles within each group. LoBoost requires no retraining, auxiliary models, or extra splitting beyond the standard train/calibration split. Experiments show competitive interval quality, improved test MSE on most datasets, and large calibration speedups.
Engineering Breakdown
Plain English
LoBoost solves the problem of uncertainty quantification in gradient-boosted trees (GBTs), which are excellent predictors for tabular data but only provide point estimates without confidence intervals. Standard conformal prediction methods use a single global residual threshold across all inputs, performing poorly when prediction error varies across the feature space (heteroscedasticity). The authors propose LoBoost, which reuses the tree ensemble's internal leaf structure to create adaptive, multiscale calibration groups—grouping similar inputs by their path through the trees—enabling distribution-free confidence intervals that are both computationally efficient and data-efficient without requiring auxiliary models or extra data splits.
Core Technical Contribution
LoBoost's core innovation is exploiting the tree structure already learned during GBT training to define local conformal calibration regions at multiple granularity levels. Rather than fitting separate nuisance models or creating additional holdout sets for score learning, the method encodes each input as its sequence of visited leaves and groups points by matching prefixes at resolution k, creating a hierarchy of calibration groups from coarse (shallow tree prefix) to fine (deep leaf). This is fundamentally different from prior local conformal methods that either use distance-based neighborhoods (expensive) or learn auxiliary regressors for adaptive scores; LoBoost achieves locality and adaptivity purely through the ensemble's existing structure, with near-zero overhead.
How It Works
The algorithm operates in two phases: offline calibration and online prediction. During calibration, LoBoost takes the fitted GBT ensemble and extracts the leaf sequence each training example visits—this is an ordered list of leaf nodes from the tree structure. At each resolution level k, it truncates these sequences to their first k nodes, creating groups of inputs that follow identical paths down the first k splits. For each group, it computes the quantile of residuals at the desired confidence level (e.g., 90th percentile). At inference time, a new input traverses the same trees, generating its leaf sequence; the method looks up which calibration group it belongs to at the chosen resolution level k and returns a prediction interval using that group's quantile. The key insight is that inputs with similar leaf sequences likely have similar residual distributions, so using these inherent groupings for conformal score calibration captures heteroscedasticity without additional model fitting.
Production Impact
For engineers deploying GBT models (XGBoost, LightGBM, CatBoost), LoBoost offers a near-zero-cost uncertainty quantification layer that can be bolted onto existing trained models. In a production system, you would add a single offline pass over your validation set to compute residual quantiles for each leaf-prefix group, then cache these thresholds—adding negligible latency to inference since leaf traversal is already happening. This enables business applications requiring confidence bands (pricing, risk modeling, anomaly detection) without retraining or holding out large calibration sets, preserving data efficiency compared to split conformal. The trade-off is that you must choose resolution level k upfront; too shallow and groups are heterogeneous (poor adaptivity), too deep and groups have few samples (unreliable quantile estimates). In practice, you'd likely validate a few k values on a holdout set—a one-time cost.
Limitations and When Not to Use This
LoBoost assumes the GBT's leaf structure meaningfully correlates with residual distribution, which may not hold if the model is severely underfit or if heteroscedasticity is driven by subtle feature interactions the trees don't capture. The method is sensitive to the choice of resolution level k, requiring validation data to tune; there is no principled, data-free way to select k automatically provided in the abstract. It also inherently produces discrete confidence intervals (due to finite leaf-prefix groups), potentially wasting samples in very sparse regions of the feature space where few examples share a deep prefix. Finally, the approach is specific to tree ensembles and does not generalize to neural networks, linear models, or other architectures—you must refit using GBTs if you want to adopt LoBoost.
Research Context
This work sits at the intersection of conformal prediction and tree-based ensemble learning. It builds on classical split conformal prediction (Lei et al.) and recent local conformal methods (e.g., localized quantile regression, distance-weighted neighborhoods) but distinguishes itself by leveraging the tree structure rather than external spatial metrics. The paper likely benchmarks against standard split conformal and other adaptive methods (local quantile forests, adaptive residual-based approaches) on tabular regression datasets (UCI, Kaggle, OpenML). It opens a research direction into structure-aware conformal methods—the idea that base models' learned representations can be mined for conformal calibration without re-training, applicable to other ensembles (random forests, boosted stumps) and potentially neural decision trees.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::