Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks

A theoretical understanding of generalization remains an open problem for many machine learning models, including deep networks where overparameterization leads to better performance, contradicting the conventional wisdom from classical statistics. Here, we investigate generalization error for kernel regression, which, besides being a popular machine learning method, also describes certain infinitely overparameterized neural networks. We use techniques from statistical mechanics to derive an analytical expression for generalization error applicable to any kernel and data distribution. We present applications of our theory to real and synthetic datasets, and for many kernels including those that arise from training deep networks in the infinite-width limit. We elucidate an inductive bias of kernel regression to explain data with simple functions, characterize whether a kernel is compatible with a learning task, and show that more data may impair generalization when noisy or not expressible by the kernel, leading to non-monotonic learning curves with possibly many peaks. Canatar et al. propose a predictive theory of generalization in kernel regression applicable to real data. This theory explains various generalization phenomena observed in wide neural networks, which admit a kernel limit and generalize well despite being overparameterized.

Automated summary

This study investigates the generalization error of kernel regression using techniques from statistical mechanics to derive an analytical expression applicable to any kernel and data distribution. The research elucidates the inductive bias of kernel regression, explains its compatibility with learning tasks, and points out that more data may impair generalization in certain cases. Canatar et al. propose a predictive theory of generalization in kernel regression that explains various phenomena observed in wide neural networks.