Journal
PROCEEDINGS OF 2019 IEEE/ACM THIRD WORKSHOP ON DEEP LEARNING ON SUPERCOMPUTERS (DLS)
Volume -, Issue -, Pages 84-94Publisher
IEEE COMPUTER SOC
DOI: 10.1109/DLS49591.2019.00016
Keywords
HPC; software deployment; performance evaluation; scalable machine learning
Funding
- U.S. Department of Energy [DE-AC05-00OR22725]
Ask authors/readers for more resources
The rapid growth and wide applicability of Deep Learning (DL) frameworks poses challenges to computing centers which need to deploy and support the software, and also to domain scientists who have to keep up with the system environment and scale up scientific exploration through DL. We offer recommendations for deploying and scaling DL frameworks on the Summit supercomputer, currently atop the Top500 list, at the Oak Ridge National Laboratory Leadership Computing Facility (OLCF). We discuss DL software deployment in the form of containers, and compare performance of native-built frameworks and containerized deployment. Software containers show no noticeable negative performance impact and exhibit faster Python loading times and promise easier maintenance. To explore strategies for scaling up DL model training campaigns, we assess DL compute kernel performance, discuss and recommend I/O data formats and staging, and identify communication needs for scalable message exchange for DL runs at scale. We recommend that users take a step-wise tuning approach beginning with algorithmic kernel choice, node I/O configuration, and communications tuning as best-practice. We present baseline examples of scaling efficiency 87% for a DL run of ResNet50 running on 1024 nodes (6144 V100 GPUs).
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available