Model compression has been introduced to reduce the required hardware resources while maintaining the model accuracy. Lots of techniques for model compression, such as pruning, quantization, and low-rank approximation, have been suggested along with different inference implementation characteristics. Adopting model compression is, however, still challenging because the design complexity of model compression is rapidly increasing due to additional hyper-parameters and computation overhead in order to achieve a high compression ratio. In this paper, we propose a simple and efficient model compression framework called DeepTwist which distorts weights in an occasional manner without modifying the underlying training algorithms. The ideas of designing weight distortion functions are intuitive and straightforward given formats of compressed weights. We show that our proposed framework improves compression rate significantly for pruning, quantization, and low-rank approximation techniques while the efforts of additional retraining and/or hyper-parameter search are highly reduced. Regularization effects of DeepTwist are also reported.
This paper proposed a general framework, DeepTwist, for model compression. The so-called weight distortion procedure is added into the training every several epochs. Three applications are shown to demonstrate the usage of the proposed approach.
Overall, I think the novelty of the paper is very limited, as all the weight distortion algorithms in the paper can be formulated as the proximal function in proximal gradient descent. See http://www.stat.cmu.edu/~ryantibs/convexopt-S15/scribes/08-prox-grad-scribed.pdf for a reference.
Specifically, the proposed framework can be easily reformulated as a loss function plus a regularizer for proximal gradient. Using gradient descent (GD), there will be two steps: (1) finding a new solution using GD, and (2) project the new solution using proximal function. Now in deep learning, since SGD is used for optimization, several steps are need to locate reasonable solutions, i.e. the Distortion Step in the framework. Then proximal function can be applied directly after Distortion Step to project the solutions. In this way, we can easily see that the proposed framework is a stochastic version of proximal gradient descent. Since SGD is used for training, several minibatches are needed to achieve a relatively stable solution for projection using the proximal function, which is exactly the proposed framework in Fig. 1.
PS: After discussion, I think the motivation of the method is not clear to understand why the proposed method works.