Journal
JOURNAL OF COMPUTATIONAL AND GRAPHICAL STATISTICS
Volume -, Issue -, Pages -Publisher
TAYLOR & FRANCIS INC
DOI: 10.1080/10618600.2023.2197488
Keywords
Bayesian inference; Langevin dynamics; Neural networks; Overparameterization
Categories
Ask authors/readers for more resources
This article discusses the estimation of the generalization gap for overparameterized models, including neural networks. It shows that the functional variance characterizes the generalization gap even in overparameterized settings. To overcome the computational cost, the Langevin approximation of the functional variance (Langevin FV) is proposed, which only relies on the first-order gradient of the squared loss function. The method is demonstrated numerically on overparameterized linear regression and nonlinear neural network models.
This article discusses the estimation of the generalization gap, the difference between generalization performance and training performance, for overparameterized models including neural networks. We first show that a functional variance, a key concept in defining a widely-applicable information criterion, characterizes the generalization gap even in overparameterized settings where a conventional theory cannot be applied. As the computational cost of the functional variance is expensive for the overparameterized models, we propose an efficient approximation of the function variance, the Langevin approximation of the functional variance (Langevin FV). This method leverages only the first-order gradient of the squared loss function, without referencing the second-order gradient; this ensures that the computation is efficient and the implementation is consistent with gradient-based optimization algorithms. We demonstrate the Langevin FV numerically by estimating the generalization gaps of overparameterized linear regression and nonlinear neural network models, containing more than a thousand of parameters therein. for this article are available online.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available