No Fine-Tuning, No Cry: Robust SVD for Compressing Deep Networks

A common technique for compressing a neural network is to compute the k-rank l2 approximation Ak of the matrix A is an element of Rnxd via SVD that corresponds to a fully connected layer (or embedding layer). Here, d is the number of input neurons in the layer, n is the number in the next one, and Ak is stored in O((n+d)k) memory instead of O(nd). Then, a fine-tuning step is used to improve this initial compression. However, end users may not have the required computation resources, time, or budget to run this fine-tuning stage. Furthermore, the original training set may not be available. In this paper, we provide an algorithm for compressing neural networks using a similar initial compression time (to common techniques) but without the fine-tuning step. The main idea is replacing the k-rank l2 approximation with lp, for p is an element of[1,2], which is known to be less sensitive to outliers but much harder to compute. Our main technical result is a practical and provable approximation algorithm to compute it for any p >= 1, based on modern techniques in computational geometry. Extensive experimental results on the GLUE benchmark for compressing the networks BERT, DistilBERT, XLNet, and RoBERTa confirm this theoretical advantage.

Automated summary

The study introduces an algorithm for compressing neural networks that utilizes modern techniques in computational geometry to approximate lp instead of k-rank l2 for effective compression. Experimental results confirm the practicality and theoretical advantage of this method in compressing networks such as BERT, DistilBERT, XLNet, and RoBERTa on the GLUE benchmark.