Takeuchi's Information Criteria as Generalization Measures for DNNs Close to NTK Regime
| Authors | Hiroki Naganuma et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2602.23219 |
| Download | |
| Categories | cs.LG |
Abstract
Generalization measures have been studied extensively in the machine learning community to better characterize generalization gaps. However, establishing a reliable generalization measure for statistically singular models such as deep neural networks (DNNs) is difficult due to their complex nature. This study focuses on Takeuchi's information criterion (TIC) to investigate the conditions under which this classical measure can effectively explain the generalization gaps of DNNs. Importantly, the developed theory indicates the applicability of TIC near the neural tangent kernel (NTK) regime. In a series of experiments, we trained more than 5,000 DNN models with 12 architectures, including large models (e.g., VGG-16), on four datasets, and estimated the corresponding TIC values to examine the relationship between the generalization gap and the TIC estimates. We applied several TIC approximation methods with feasible computational costs and assessed the accuracy trade-off. Our experimental results indicate that the estimated TIC values correlate well with the generalization gap under conditions close to the NTK regime. However, we show both theoretically and empirically that outside the NTK regime such correlation disappears. Finally, we demonstrate that TIC provides better trial pruning ability than existing methods for hyperparameter optimization.
Engineering Breakdown
Plain English
This paper investigates Takeuchi's Information Criterion (TIC), a classical statistical measure, as a way to predict how well deep neural networks will generalize to unseen data. The authors trained over 5,000 DNN models across 12 different architectures (including VGG-16) on four datasets and computed TIC values to test whether this measure can reliably explain generalization gaps. The key finding is that TIC becomes an effective generalization measure specifically when DNNs operate near the Neural Tangent Kernel (NTK) regime, a theoretical regime where neural networks behave like infinite-width kernel methods. This provides a principled, theoretically-grounded way to estimate generalization performance for deep networks without requiring expensive validation splits or holdout data.
Core Technical Contribution
The paper's core contribution is establishing the theoretical and empirical conditions under which Takeuchi's Information Criterion—originally developed for classical statistical models—can reliably measure generalization in deep neural networks. Unlike prior generalization bounds that tend to be vacuous or loosely connected to actual test error, TIC provides a data-dependent, practically computable measure that correlates with real generalization gaps when the network is close to the NTK regime. The authors prove this connection theoretically and validate it empirically across 5,000+ trained models, moving beyond toy problems to realistic architectures. This bridges classical statistical theory (TIC) with modern deep learning, offering a concrete alternative to existing complexity measures like Rademacher complexity or PAC-Bayes bounds.
How It Works
Takeuchi's Information Criterion extends the Akaike Information Criterion (AIC) to handle model misspecification by using the full Hessian of the loss function rather than assuming the model is correctly specified. For a trained DNN, TIC is computed as: TIC = Loss(θ) + tr(H^{-1}V), where H is the Hessian of the training loss, V is the covariance of the gradient of the loss at the optimal parameters θ, and tr denotes the trace. In the NTK regime (infinite-width or sufficiently wide networks), the neural network kernel becomes fixed during training and behaves like a linear model with respect to its parameters, making H^{-1}V interpretable and predictive of generalization. The authors estimate these quantities on trained models, then compare the TIC values to actual test error across the 5,000 models to verify the measure's validity. They also examine how this relationship holds or breaks down as networks move away from the NTK regime (e.g., in feature-learning regimes with smaller widths).
Production Impact
For practitioners building production ML systems, this work enables better model selection and hyperparameter tuning without costly cross-validation or separate test sets. Instead of splitting data for validation (losing training capacity), engineers could compute TIC on the training set to predict generalization, enabling use of more data for training in data-constrained settings. This is particularly valuable for large models where validation splits are expensive—computing TIC requires backpropagation to get the Hessian and gradient covariance, which is feasible at training time but still requires careful implementation of second-order derivatives. The trade-off is computational: TIC requires Hessian computation (typically O(n²) memory for n parameters), making it expensive for very large modern models; you'd need efficient approximations or low-rank estimates in practice. For models near the NTK regime (relatively fixed features, wide networks, short training), TIC offers a reliable, principled alternative to ad-hoc generalization estimates.
Limitations and When Not to Use This
The paper's applicability is fundamentally limited to the NTK regime or near-NTK conditions—real modern deep learning (vision transformers, large language models) operates predominantly in the feature-learning regime where networks learn and evolve their representations significantly, making TIC's theoretical guarantees potentially invalid. Computing the full Hessian is computationally prohibitive for models with millions or billions of parameters, requiring either matrix-free approximations (which add complexity and approximation error) or restriction to smaller models. The empirical validation uses relatively small datasets (four datasets, mostly image classification) and older architectures (VGG-16 as the largest); it remains unclear whether the findings scale to modern, overparameterized networks like ResNets with batch norm or transformer architectures. The paper doesn't address how TIC behaves under common practical conditions like stochastic gradient descent with momentum, data augmentation, or regularization techniques that may shift the effective training dynamics away from the NTK regime.
Research Context
This work builds on decades of research into generalization theory for neural networks, including classical PAC-Bayes bounds, margin-based complexity measures, and recent advances in understanding the Neural Tangent Kernel regime (introduced by Jacot et al. 2018). It leverages the NTK framework as a tractable theoretical setting where neural network training dynamics are well-understood and amenable to classical statistical tools. The paper directly addresses a gap in the literature: most existing generalization bounds are too loose to be practically useful for real models, while empirical measures (early stopping, validation error) lack theoretical grounding. This opens a research direction toward bringing classical statistical inference tools (like information criteria) into the deep learning setting, potentially inspiring follow-up work on other classical measures (BIC, cross-validation estimators) adapted for neural networks in well-characterized regimes.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::