ENERGY-CONSTRAINED COMPRESSION FOR DEEP NEURAL NETWORKS VIA WEIGHTED SPARSE PROJEC- TION AND LAYER INPUT MASKING
Deep Neural Networks (DNNs) are increasingly deployed in highly energyconstrained environments such as autonomous drones and wearable devices while at the same time must operate in real-time. Therefore, reducing the energy consumption has become a major design consideration in DNN training. This paper proposes the first end-to-end DNN training framework that provides quantitative energy consumption guarantees via weighted sparse projection and input masking. The key idea is to formulate the DNN training as an optimization problem in which the energy budget imposes a previously unconsidered optimization constraint. We integrate the quantitative DNN energy estimation into the DNN training process to assist the constrained optimization. We prove that an approximate algorithm can be used to efficiently solve the optimization problem. Compared to the best prior energy-saving methods, our framework trains DNNs that provide higher accuracies under same or lower energy budgets.
This paper describes a procedure for training neural networks via an explicit constraint on the energy budget, as opposed to pruning the model size as commonly done with standard compression methods.  Comparative results are shown on a few data sets where the proposed method outperforms multiple different approaches.  Overall, the concept is interesting and certainly could prove valuable in resource-constrained environments.  Still I retain some reservations as detailed below.
My first concern is that this paper exceeds the recommended 8 page limit for reasons that are seemingly quite unnecessary.  There are no large, essential figures/tables, and nearly the first 6 pages is just introduction and background material.  Likewise the paper consumes a considerable amount of space presenting technical results related to knapsack problems and various epsilon-accurate solutions, but this theoretical content seems somewhat irrelevant and distracting since it is not directly related to the greedy approximation strategy actually used for practical deployment.  Much of this material could have been moved to the supplementary so as to adhere to the 8 page soft limit.  Per the ICLR reviewer instructions, papers deemed unnecessarily long relative to this length should be judged more critically.
Another issue relates to the use of a mask for controlling the sparsity of network inputs.  Although not acknowledged, similar techniques are already used to prune the activations of deep networks for compression.  In particular, various forms of variational dropout essentially use multiplicative weights to remove the influence of activations and/or other network components similar to the mask M used is this work.  Representative examples include Neklyudov et al., "Structured Bayesian Pruning via Log-Normal Multiplicative Noise," NIPS 2017 and Louizos et al., "Bayesian Compression for Deep Learning," NIPS 2017, but there are many other related alternatives using some form of trainable gate or mask, possibly stochastic, to affect pruning (the major ML and CV conferences over the past year have numerous related compression papers).  So I don't consider this aspect of the paper to be new in any significant way.
Moreover, for the empirical comparisons it would be better to compare against state-of-the-art compression methods as opposed to just the stated MP and SSL methods from 2015 and 2016 respectively.  Despite claims to the contrary on page 9, I would not consider these to be state-of-the-art methods at this point.
Another comment I have regarding the experiments is that hyperparameters and the use of knowledge distillation were potentially tuned for the proposed method and then simultaneously applied to the competing algorithms for the sake of head-to-head comparison.  But to me, if these enhancements are to be included at all, tuning must be done carefully and independently for each algorithm.  Was this actually done?  Moreover it would have been nice to see results without the confounding influence of distillation to isolate sources of improvement, but no ablation studies were presented.
Finally, regarding the content in Section 5, the paper carefully presents an explicit bound on energy that ultimately leads to a constraint that is NP-hard just to project on to, although approximate solutions exist that depend on some error tolerance.  However, even this requires an algorithm that is dismissed as "complicated."  Instead a greedy alternative is derived in the Appendix which presumably serves as the final endorsed approach.  But at this point it is no longer clear to me exactly what performance guarantees remain with respect to the energy bound.  Theorem 3 presents a fairly inscrutable bound, and it is not at all transparent how to interpret this in any practical sense.  Note that after Theorem 3, conditions are described whereby an optimal projection can be obtained, but these seem highly nuanced, and unlikely to apply in most cases.
Additionally, it would appear that crude bounds on the energy could also be introduced by simply penalizing/constraining the sparsity on each layer, which leads to a much simpler projection step.  For example, a simple affine function of the L0 norm would be much easier to optimize and could serve as a loose bound on the energy, given that the latter should be a non-decreasing function of the L0 norm.  Any idea how such a bound compares to those presented given all the approximations and greedy steps that must be included?
Other comments:
- As an implementation heuristic, the proposed Algorithm 1 gradually decays the parameter q, which controls the sparsity of the mask M.  But this will certainly alter the energy budget, and I wonder how important it is to employ a complex energy constraint if minimization requires this type of heuristic.
- I did not see where the quantity L(M,W) embedded in eq. (17) was formally defined, although I can guess what it is.
- In general it is somewhat troublesome that, on top of a complex, non-convex deep network energy function, just the small subproblem required for projecting onto the energy constraint is NP-hard.  Even if approximations are possible, I wonder if this extra complexity is always worth it relative so simple sparsity-based compression methods which can be efficiently implemented with exactly closed-form projections.
- In Table 1, the proposed method is highlighted as having the smallest accuracy drop on SqueezeNet.  But this is not true, EAP is lower.  Likewise on AlexNet, NetAdapt has an equally optimal energy.