Structured Bayesian Pruning via Log-Normal Multiplicative Noise
Dropout-based regularization methods can be regarded as injecting random noise with pre-defined magnitude to different parts of the neural network during training. It was recently shown that Bayesian dropout procedure not only improves generalization but also leads to extremely sparse neural architectures by automatically setting the individual noise magnitude per weight. However, this sparsity can hardly be used for acceleration since it is unstructured. In the paper, we propose a new Bayesian model that takes into account the computational structure of neural networks and provides structured sparsity, e.g. removes neurons and/or convolutional channels in CNNs. To do this we inject noise to the neurons outputs while keeping the weights unregularized. We establish the probabilistic model with a proper truncated log-uniform prior over the noise and truncated log-normal variational approximation that ensures that the KL-term in the evidence lower bound is computed in closed-form. The model leads to structured sparsity by removing elements with a low SNR from the computation graph and provides significant acceleration on a number of deep neural architectures. The model is easy to implement as it can be formulated as a separate dropout-like layer.
employs group-wise sparsity in convolutional filters, Perforated CNNs [3] drop redundant rows from the intermediate dataframe matrices that are used to compute convolutions, and Structured Sparsity Learning [24] provides a way to remove entire convolutional filters or even layers in residual networks. These methods allow to obtain practical acceleration with little to no modifications of the existing software. In this paper, we propose a tool that is able to induce an arbitrary pattern of structured sparsity on neural network parameters or intermediate data tensors. We propose a dropout-like layer with a parametric multiplicative noise and use stochastic variational inference to tune its parameters in a Bayesian way. We introduce a proper analog of sparsity-inducing log-uniform prior distribution [8,16] that allows us to formulate a correct probabilistic model and avoid the problems that come from using an improper prior. This way we obtain a novel Bayesian method of regularization of neural networks that results in structured sparsity. Our model can be represented as a separate dropout-like layer that allows for a simple and flexible implementation with almost no computational overhead, and can be incorporated into existing neural networks.
Our experiments show that our model leads to high group sparsity level and significant acceleration of convolutional neural networks with negligible accuracy drop. We demonstrate the performance of our method on LeNet and VGG-like architectures using MNIST and CIFAR-10 datasets.
Summary:
The paper addresses the actual problem of compression and efficient computation of deep NNs. It presents a new pruning technique for deep NNs, which is an enhancement of the Bayesian dropout method. Contrary to standard Bayesian dropout, the proposed method prunes NNs in a structured way. The NN-model is altered by dropout layers with an advanced probabilistic model (with truncated log-uniform prior distribution and truncated log-uniform posterior distribution). For the altered model, the authors propose a new "signal-to-ratio" pruning metric applicable to both individual neurons and more complex structures (e.g., filters). 
The submission and supplementary files contain a theoretical explanation of the method and derivation of the corresponding formulas. Moreover, an experimental evaluation of the approach is presented, with promising results. 
Quality:
The paper has very good technical quality. It contains a thorough theoretical analysis of the proposed model and its properties. It also presents several experimental results, that evaluate the qualities of the method and compare it with two concurrent approaches.
Clarity:
The presentation is comprehensible and well-organized, the method is described in detail including derivation of formulas and implementation details. 
Originality and significance:
The paper addresses the actual problem of compression and efficient computation of deep NNs. Its main contribution is the presentation of a new general technique for structured pruning of NNs that is based on Bayesian dropout. Contrary to the original method, 
1) it uses more sophisticated and proper noise distributions and it involves a novel pruning metric,
2) the subject of pruning are not weights, but more complex structures (e.g., filters, neurons).
3) The method can be applied to several NN architectures (CNN, fully-connected,...), and it seems that it can be easily added to existing NN implementations.
The presented approach proved in the experiments to massively prune the network and to speed up the computation on GPU and CPU, while inducing only small accuracy loss. However, based on the experimental results presented, the benefit over the concurrent "SSL" approach is relatively small.
Typo:
...that is has ... (line 117)
Comments:
- Table 1 and Table 2 contain rows with results for concurrent methods called "SSL" and "SparseVD", without a short description of the main principles of these methods in the text.
- It would be useful to assess also the efficiency of the pruning process itself.
- The method has several parameters that need to be set. An interesting question is, how does the actual setting of these parameters affect the balance between sparsity and accuracy of the final model.