期刊
NEUROCOMPUTING
卷 394, 期 -, 页码 41-50出版社
ELSEVIER
DOI: 10.1016/j.neucom.2020.01.106
关键词
Neural networks; Learning rates; LSTM; Optimization; Gradients
资金
- National Natural Science Foundation of China [61772124]
In the training process of the deep neural networks, the learning rate plays an important role in whether the training process can converge and how fast it can achieve converge. In order to ensure convergence, most of the existing optimization methods adopt a multi-stage descending small learning rate which is hand-designed. However, this method converges slowly especially in the early stage of training. Based on this, a learning rate adjustment strategy that can automatically adjust and has a faster speed of loss decline will be helpful to the training of the deep model. In this paper, a dynamic adjustment strategy of learning rate is developed based on the Long Short Term Memory(LSTM) model and the gradients of loss function. This method effectively utilizes the advantages of the LSTM model considering the multistep learning rate as a whole, and generates the learning rate of the current step based on the memory information of the previous learning rate. Three datasets and four architectures are used in the experiments. We applied the learning rate adjustment method to various optimization methods and achieved good results that our method can achieve even smaller loss under the same number of iterations. (C) 2020 Elsevier B.V. All rights reserved.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据