GPU Tensor Cores for Fast Arithmetic Reductions

This article proposes a parallel algorithm for computing the arithmetic reduction of n numbers as a set of matrix-multiply accumulate (MMA) operations that are executed simultaneously by GPU tensor cores. The analysis, assuming tensors of size m x m, shows that the proposed algorithm has a parallel running time of T(n) = 5log(m2n) and a speedup of S = 4/5 log(2) m(2) over a canonical parallel reduction. Experimental performance results on a Tesla V100 GPU show that the tensor-core based approach is energy efficient and runs up to similar to 3.2x and 2x faster than a standard GPU-based reduction and Nvidia's CUB library, respectively, while keeping the numerical error below 1 percent with respect to a double precision CPU reduction. The chained design of the algorithm allows a flexible configuration of GPU thread-blocks and the optimal values found through experimentation agree with the theoretical ones. The results obtained in this work show that GPU tensor cores are relevant not only for Deep Learning or Linear Algebra computations, but also for applications that require the acceleration of large summations.

Automated summary

This article introduces a parallel algorithm for arithmetic reduction using GPU tensor cores, achieving faster performance and energy efficiency. Experimental results demonstrate that the proposed method outperforms standard GPU reduction and Nvidia's CUB library by approximately 3.2x and 2x, respectively, while maintaining low numerical error.