期刊
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
卷 44, 期 1, 页码 404-415出版社
IEEE COMPUTER SOC
DOI: 10.1109/TPAMI.2020.3004354
关键词
Natural gradient descent; distributed deep learning; deep convolutional neural networks; image classification
资金
- ABCI Grand Challenge Program, National Institute of Advanced Industrial Science and Technology (AIST)
- JSPS KAKENHI [JP18H03248, JP19J13477]
- JST CREST, Japan [JPMJCR19F5]
- Joint Usage/Research Center for Interdisciplinary Large-scale Information Infrastructures in Japan [jh180012-NAHI]
- Tokyo Tech through the HPCI System Research Project [hp190122]
This paper proposes a scalable and practical natural gradient descent (SP-NGD) method for large-scale distributed training of deep neural networks. It achieves similar generalization performance to models trained with first-order optimization methods, but with accelerated convergence.
Large-scale distributed training of deep neural networks results in models with worse generalization performance as a result of the increase in the effective mini-batch size. Previous approaches attempt to address this problem by varying the learning rate and batch size over epochs and layers, or ad hoc modifications of batch normalization. We propose scalable and practical natural gradient descent (SP-NGD), a principled approach for training models that allows them to attain similar generalization performance to models trained with first-order optimization methods, but with accelerated convergence. Furthermore, SP-NGD scales to large mini-batch sizes with a negligible computational overhead as compared to first-order methods. We evaluated SP-NGD on a benchmark task where highly optimized first-order methods are available as references: training a ResNet-50 model for image classification on ImageNet. We demonstrate convergence to a top-1 validation accuracy of 75.4 percent in 5.5 minutes using a mini-batch size of 32,768 with 1,024 GPUs, as well as an accuracy of 74.9 percent with an extremely large mini-batch size of 131,072 in 873 steps of SP-NGD.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据