Scalar Quantization as Sparse Least Square Optimization

Quantization aims to form new vectors or matrices with shared values close to the original. In recent years, the popularity of scalar quantization has been soaring as it is found huge utilities in reducing the resource cost of neural networks. Popular clustering-based techniques suffers substantially from the problems of dependency on the seed, empty or out-of-the-range clusters, and high time complexity. To overcome the problems, in this paper, scalar quantization is examined from a new perspective, namely sparse least square optimization. Specifically, several quantization algorithms based on l(1) least square are proposed and implemented. In addition, similar schemes with l(1) + l(2) and l(0) regularization are proposed. Furthermore, to compute quantization results with given amount of values/clusters, this paper proposes an iterative method and a clustering-based method, and both of them are built on sparse least square optimization. The algorithms proposed are tested under three data scenarios and their computational performance, including information loss, time consumption, and distribution of values of sparse vectors are compared. The paper offers a new perspective to probe the area of quantization, and the algorithms proposed are superior especially under bit-width reduction scenarios, where the required post-quantization resolution (the number of values) is not significantly lower than the original scalar.

Automated summary

This paper investigates the application of scalar quantization based on sparse least square optimization, proposing multiple quantization algorithms based on different regularization criteria, and comparing and testing them through iterative methods and clustering-based methods.