A Quantitative Survey of Communication Optimizations in Distributed Deep Learning

Nowadays, large and complex deep learning (DL) models are increasingly trained in a distributed manner across multiple worker machines, in which extensive communications between workers pose serious scaling problems. In this article, we present a quantitative survey of communication optimization techniques for data parallel distributed DL. We first identify the major communication challenges and classify the existing solutions into three levels, namely the learning algorithm, the system architecture, and the network infrastructure. We present the state-of-the-art communication optimization techniques and conduct a comparative study of seven common lossless distributed DL methods on a 32-GPU cluster with 100Gb/s InfiniBand (IB). We show that the DL models with low model intensity (such as BERT and BERT-Large) are difficult to scale out even with the best available lossless algorithm over 100Gb/s IB; and the system architecture and scheduling algorithms have a critical impact on the scaling property. We conclude the article with discussions of open issues for further investigation.

Automated summary

This article presents a quantitative survey of communication optimization techniques for data parallel distributed deep learning, identifying major challenges and classifying solutions at different levels. A comparative study of seven common methods on a 32-GPU cluster with 100Gb/s InfiniBand reveals the difficulties in scaling DL models with low model intensity, and highlights the critical impact of system architecture and scheduling algorithms on performance. Discussions on open issues for further investigation are also provided.