期刊
NEUROCOMPUTING
卷 450, 期 -, 页码 354-361出版社
ELSEVIER
DOI: 10.1016/j.neucom.2021.04.047
关键词
Deep neural network; Dropout; Network training; Sparsity
资金
- National Natural Science Foundation of China [61702349]
- National Key Research and Development Program of China [2018AAA0100201]
AS-Dropout is a new adaptive sparse dropout method that can learn both certainty and uncertainty in neural network training, increasing network sparsity. Experimental results show that AS-Dropout substantially outperforms traditional dropout and some improved methods on MNIST, COIL-100, and Caltech-101 datasets.
Dropout is an important training method for deep neural networks, because it can help avoid over-fitting. Traditional dropout methods and many extended dropout methods, omit some of the neurons' activation values according to the probabilities. These methods calculate the activation probability of neurons using the designed formula, without providing a plausible explanation of the calculation method. This paper proposes an adaptive sparse dropout (AS-Dropout) method for neural network training. The algorithm maps the neurons' activation values in a layer to a relative linear range of a sigmoid function, determines the ratio of active neurons by a probability calculation process, and drops most of neurons according to the probabilities. The probability calculation depends on the activation values of the neurons. The selection of active neurons is according to the probabilities. Therefore, AS-Dropout learns both the certainty and uncertainty in deep neural networks. Additionally, since only a small number of neurons are active, AS-Dropout increases the sparsity of the network. We applied AS-Dropout in different neural network structures. When evaluated on MNIST, COIL-100, and Caltech-101 datasets, the experimental results demonstrated that, overall, AS-Dropout substantially outperformed the traditional dropout and some improved dropout methods. CO 2021 Published by Elsevier B.V.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据