A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification

Recent advancements in machine learning achieved by Deep Neural Networks (DNNs) have been significant. While demonstrating high accuracy, DNNs are associated with a huge number of parameters and computations, which leads to high memory usage and energy consumption. As a result, deploying DNNs on devices with constrained hardware resources poses significant challenges. To overcome this, various compression techniques have been widely employed to optimize DNN accelerators. A promising approach is quantization, in which the full-precision values are stored in low bit-width precision. Quantization not only reduces memory requirements but also replaces high-cost operations with low-cost ones. DNN quantization offers flexibility and efficiency in hardware design, making it a widely adopted technique in various methods. Since quantization has been extensively utilized in previous works, there is a need for an integrated report that provides an understanding, analysis, and comparison of different quantization approaches. Consequently, we present a comprehensive survey of quantization concepts and methods, with a focus on image classification. We describe clustering-based quantization methods and explore the use of a scale factor parameter for approximating full-precision values. Moreover, we thoroughly review the training of a quantized DNN, including the use of a straight-through estimator and quantization regularization. We explain the replacement of floating-point operations with low-cost bitwise operations in a quantized DNN and the sensitivity of different layers in quantization. Furthermore, we highlight the evaluation metrics for quantization methods and important benchmarks in the image classification task. We also present the accuracy of the state-of-the-art methods on CIFAR-10 and ImageNet. This article attempts to make the readers familiar with the basic and advanced concepts of quantization, introduce important works in DNN quantization, and highlight challenges for future research in this field.

Automated summary

Recent advancements in machine learning achieved by Deep Neural Networks (DNNs) have led to significant improvements in accuracy; however, the high number of parameters and computations associated with DNNs result in high memory usage and energy consumption. To address this issue, various compression techniques have been employed, with quantization being a promising approach. This article presents a comprehensive survey of quantization concepts and methods, focusing on image classification. It covers clustering-based quantization methods, the use of scale factor parameter, training of a quantized DNN, replacement of floating-point operations with bitwise operations, sensitivity of different layers in quantization, evaluation metrics, and benchmarks. The article aims to familiarize readers with quantization concepts, introduce important works in DNN quantization, and highlight challenges for future research in this field.