期刊
IEEE TRANSACTIONS ON SIGNAL PROCESSING
卷 65, 期 16, 页码 4265-4280出版社
IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
DOI: 10.1109/TSP.2017.2708039
关键词
Deep learning; deep neural networks; generalization error; robustness
资金
- Engineering and Physical Sciences Research Council [Ep/K033166/1]
- German-Israeli Foundation for Scientific Research and Development (GIF)
- National Science Foundation
- Office of Naval Research
- ARO
- NGA
- EPSRC [EP/K033166/1] Funding Source: UKRI
- Engineering and Physical Sciences Research Council [EP/K033166/1] Funding Source: researchfish
The generalization error of deep neural networks via their classification margin is studied in this paper. Our approach is based on the Jacobian matrix of a deep neural network and can be applied to networks with arbitrary nonlinearities and pooling layers, and to networks with different architectures such as feed forward networks and residual networks. Our analysis leads to the conclusion that a bounded spectral norm of the network's Jacobian matrix in the neighbourhood of the training samples is crucial for a deep neural network of arbitrary depth and width to generalize well. This is a significant improvement over the current bounds in the literature, which imply that the generalization error grows with either the width or the depth of the network. Moreover, it shows that the recently proposed batch normalization and weight normalization reparametrizations enjoy good generalization properties, and leads to a novel network regularizer based on the network's Jacobian matrix. The analysis is supported with experimental results on the MNIST, CIFAR-10, LaRED, and ImageNet datasets.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据