Bayesian Compression for Deep Learning
Compression and computational efficiency in deep learning have become a problem of great significance. In this work, we argue that the most principled and effective way to attack this problem is by adopting a Bayesian point of view, where through sparsity inducing priors we prune large parts of the network. We introduce two novelties in this paper: 1) we use hierarchical priors to prune nodes instead of individual weights, and 2) we use the posterior uncertainties to determine the optimal fixed point precision to encode the weights. Both factors significantly contribute to achieving the state of the art in terms of compression rates, while still staying competitive with methods designed to optimize for speed or energy efficiency.
This paper describes a method for compressing a neural network: pruning its neurons and reducing the floating point precision of its weights to make it smaller and faster at test time. The method works by training the neural net with SVI using a factored variational approximation that shares a scale parameter between sets of weights. The entropy term of the variational objective incentivizes weights to have a large variance, which leads to compression by two separate mechanisms: neurons with high-variance weights relative to their means can be pruned, and layers can have their floating point precision reduced to a level commensurate with the variance of their weights. It's a fairly principled approach in which compression arises naturally as a consequence of Bayesian uncertainty. It also seems effective in practice.
I have three main reasons for not giving a higher score. First, I'm not convinced that the work is hugely original. It leans very heavily on previous work, particularly Kingma et al. on variational dropout and Molchanov et al. on sparsification arising from variational dropout. The two novelties are (1) hierarchical priors and (2) reduced floating point precision, which are certainly useful but they are unsurprising extensions of the previous work. Additionally, the paper does a poor job of placing itself in the context of these previous papers. Despite repeatedly referencing the variational dropout papers for forms of variational approximations or derivations, the concept of 'variational dropout' is never actually introduced! The paper would be much clearer if it at least offered a few-sentence summary of each of these previous papers, and described where it departs from them.
Second, the technical language is quite sloppy in places. The authors regularly use the phrase "The KL divergence between X and Y" instead of "the KL divergence from X to Y". "Between" just shouldn't be used because it implies symmetry. Even worse, in all cases, the order of "X" and "Y" is backwards, so that even "from X to Y" would be incorrect. Another example of sloppy language is the use of the term "posterior" to describe the variational approximation q. This causes genuine confusion, because the actual posterior is a very relevant object. At the very least, the term should be "posterior approximation". In the paper's defense, the actual equations are generally clear and unambiguous.
Finally, the experiments are a bit unimaginative. For example, although the authors argue that Bayesian inference and model compression are well aligned, we know they have quite different goals and you would expect to find some situations where they could be in conflict. Why not explore this tradeoff by looking at performance and compression as a function of hyperparameters like $\tau_0$?  How much compression do you get if you select $\tau_0$ by maximizing the marginal likelihood estimate rather than choosing an arbitrary value?