Spartan: A Sparsity-Adaptive Framework to Accelerate Deep Neural Network Training on GPUs

Deep Neural Networks (DNNs) have emerged as an important class of machine learning algorithms, providing accurate solutions to a broad range of applications. Sparsity in activation maps in DNN training presents an opportunity to reduce computations. However, exploiting activation sparsity presents two major challenges: i) profiling activation sparsity during training comes with significant overhead due to computing the degree of sparsity and the data movement; ii) the dynamic nature of activation maps requires dynamic dense-to-sparse conversion during training, leading to significant overhead. In this article, we present Spartan, a lightweight hardware/software framework to accelerate DNN training on a GPU. Spartan provides a cost-effective and programmer-transparent microarchitectural solution to exploit activation sparsity detected during training. Spartan provides an efficient sparsity monitor, a tile-based sparse GEMM algorithm, and a novel compaction engine designed for GPU workloads. Spartan can reduce sparsity profiling overhead by 52.5x on average. For the most compute-intensive layers, i.e., convolutional layers, we can speedup AlexNet by 3.4x, VGGNet-16 by 2.14x, and ResNet-18 by 2.02x, when training on the ImageNet dataset.

Automated summary

The article presents Spartan, a lightweight hardware/software framework to accelerate DNN training on GPU by exploiting activation sparsity, reducing computations and improving efficiency. Spartan provides efficient tools such as sparsity monitor, sparse GEMM algorithm, and compaction engine, achieving significant reductions in sparsity profiling overhead and speeding up training on compute-intensive layers like convolutional layers.