Deep learning is becoming more widespread in its application due to its power in solving complex classification problems. However, deep learning models often require large memory and energy consumption, which may prevent them from being deployed effectively on embedded platforms, limiting their applications. This work addresses the problem by proposing methods Weight Reduction Quantisation for compressing the memory footprint of the models, including reducing the number of weights and the number of bits to store each weight. Beside, applying with sparsity-inducing regularization, our work focuses on speeding up stochastic variance reduced gradients (SVRG) optimization on non-convex problem. Our method that minibatch SVRG with 1 regularization on non-convex problem has faster and smoother convergence rates than SGD by using adaptive learning rates. Experimental evaluation of our approach uses MNIST and CIFAR-10 datasets on LeNet-300-100 and LeNet-5 models, showing our approach can reduce the memory requirements both in the convolutional and fully connected layers by up to 60× without affecting their test accuracy.
The authors present an l-1 regularized SVRG based training algorithm that is able to force many weights of the network to be 0, hence leading to good compression of the model.  The motivation for l-1 regularization is clear as it promotes sparse models, which lead to lower storage overheads during inference. The use of SVRG is motivated by the fact that it can, in some cases, provide faster convergence than SGD.
Unfortunately, the authors do not compare with some key literature. For example there has been several techniques that use sparsity, and group sparsity [1,2,3], that lead to the same conclusion as the paper here: models can be significantly sparsified while not affecting the test accuracy of the trained model.
Then, the novelty of the technique presented is also unclear, as essentially the algorithm is simply SVRG with l1 regularization and then some quantization. The experimental evaluation does not strongly support the thesis that the presented algorithm is much better than SGD with l1 regularization. In the presented experiments, the gap between the performance of SGD and SVRG is small (especially in terms of test error), and overall the savings in terms of the number of weights is similar to Deep compression. Hence, it is unclear how the use of SVRG over SGD improves things. Eg in figure 2 the differences in top-1 error of SGD and SVRG, for the same number of weights is very similar (it’s unclear also why Fig 2a uses top-1 and Fig 2b uses top-5 error). I also want to note that all experiments were run on LeNet, and not on state of the art models (eg ResNets).
Finally, the paper is riddled with typos. I attach below some of the ones I found in pages 1 and 2
Overall, although the topic is very interesting, the contribution of this paper is limited, and it is unclear how it compares with other similar techniques that use group sparsity regularization, and whether SVRG offers any significant advantages over l1-SGD.
typos:
“ This work addresses the problem by proposing methods Weight Reduction Quantisation”
-> This work addresses the problem by proposing a Weight Reduction Quantisation method
“Beside, applying with sparsity-inducing regularization”
-> Beside, applying sparsity-inducing regularization
“Our method that minibatch SVRG with l-1 regularization on non-convex problem”
-> Our minibatch SVRG with l-1 regularization method on non-convex problem
“As well as providing,l1 regularization is a powerful compression techniques to penalize some weights to be zero”
-> “l1 regularization is a powerful compression technique that forces some weights to be zero”
 The problem 1 can
->  The problem in Eq.(1) can
“it inefficiently encourages weight”
-> “it inefficiently encourages weights”
————
[1] Learning Structured Sparsity in Deep Neural Networks
http://papers.nips.cc/paper/6504-learning-structured-sparsity-in-deep-neural-networks.pdf
[2] Fast ConvNets Using Group-wise Brain Damage
https://arxiv.org/pdf/1506.02515.pdf
[3] Sparse Convolutional Neural Networks
https://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Liu_Sparse_Convolutional_Neural_2015_CVPR_paper.pdf