l1 Regularization in Two-Layer Neural Networks

A crucial problem of neural networks is to select an architecture that strikes appropriate tradeoffs between underfitting and overfitting. This work shows that l(1) regularizations for two-layer neural networks can control the generalization error and sparsify the input dimension. In particular, with an appropriate l(1) regularization on the output layer, the network can produce a tight statistical risk. Moreover, an appropriate l(1) regularization on the input layer leads to a risk hound that does not involve the input data dimension. The results also indicate that training a wide neural network with a suitable regularization provides an alternative bias-variance tradeoff to selecting from a candidate set of neural networks. Our analysis is based on a new integration of dimension-based and norm-based complexity analysis to bound the generalization error.

Automated summary

A crucial problem in neural networks is selecting an architecture that balances the tradeoff between underfitting and overfitting. This study demonstrates that l(1) regularizations for two-layer neural networks can control generalization error and sparsify input dimensions. By applying an appropriate l(1) regularization on the output layer, the network can produce a tight statistical risk. Additionally, using l(1) regularization on the input layer results in a risk constraint that is not dependent on the input data dimension. The findings also suggest that training a wide neural network with suitable regularization offers an alternative bias-variance tradeoff over selecting from a candidate set of neural networks.