Journal
IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS
Volume 31, Issue 1, Pages 94-110Publisher
IEEE COMPUTER SOC
DOI: 10.1109/TPDS.2019.2928289
Keywords
Performance evaluation; GPU; interconnect; NUMA; PCIe; NVLink; NVSwitch; SLI; GPUDirect; RDMA; NCCL
Funding
- Application Assessment program within the Exascale Computing Project of the U.S. Department of Energy Office of Science [17-SC-20-SC]
- U.S. Department of Energy Office of Science
- National Nuclear Security Administration
- U.S. DOE Office of Science, Office of Advanced Scientific Computing Research [66150]
- Office of Science of the U.S. Department of Energy [DE-AC05-00OR22725]
- U.S. Department of Energy [DE-AC05-76RL01830]
- High Performance Data Analytics (HPDA) program at PNNL
Ask authors/readers for more resources
High performance multi-GPU computing becomes an inevitable trend due to the ever-increasing demand on computation capability in emerging domains such as deep learning, big data and planet-scale simulations. However, the lack of deep understanding on how modern GPUs can be connected and the real impact of state-of-the-art interconnect technology on multi-GPU application performance become a hurdle. In this paper, we fill the gap by conducting a thorough evaluation on five latest types of modern GPU interconnects: PCIe, NVLink-V1, NVLink-V2, NVLink-SLI and NVSwitch, from six high-end servers and HPC platforms: NVIDIA P100-DGX-1, V100-DGX-1, DGX-2, OLCF's SummitDev and Summit supercomputers, as well as an SLI-linked system with two NVIDIA Turing RTX-2080 GPUs. Based on the empirical evaluation, we have observed four new types of GPU communication network NUMA effects: three are triggered by NVLink's topology, connectivity and routing, while one is caused by PCIe chipset design issue. These observations indicate that, for an application running in a multi-GPU node, choosing the right GPU combination can impose considerable impact on GPU communication efficiency, as well as the application's overall performance. Our evaluation can be leveraged in building practical multi-GPU performance models, which are vital for GPU task allocation, scheduling and migration in a shared environment (e.g., AI cloud and HPC centers), as well as communication-oriented performance tuning.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available