Published as a conference paper at ICLR 2018 LEARNING SPARSE NEURAL NETWORKS THROUGH L 0 REGULARIZATION
We propose a practical method for L 0 norm regularization for neural networks: pruning the network during training by encouraging weights to become exactly zero. Such regularization is interesting since (1) it can greatly speed up training and inference, and (2) it can improve generalization. AIC and BIC, well-known model selection criteria, are special cases of L 0 regularization. However, since the L 0 norm of weights is non-differentiable, we cannot incorporate it directly as a regularization term in the objective function. We propose a solution through the inclusion of a collection of non-negative stochastic gates, which collectively determine which weights to set to zero. We show that, somewhat surprisingly, for certain distributions over the gates, the expected L 0 regularized objective is differentiable with respect to the distribution parameters. We further propose the hard concrete distribution for the gates, which is obtained by "stretching" a binary concrete distribution and then transforming its samples with a hard-sigmoid. The parameters of the distribution over the gates can then be jointly optimized with the original network parameters. As a result our method allows for straightforward and efficient learning of model structures with stochastic gradient descent and allows for conditional computation in a principled way. We perform various experiments to demonstrate the effectiveness of the resulting approach and regularizer.
Learning sparse neural networks through L0 regularisation
Summary: 
The authors introduce a gradient-based approach to minimise an objective function with an L0 sparse penalty. The problem is relaxed onto a continuous optimisation by changing an expectation over discrete variables (representing whether a variable is present or not) to an expectation over continuous variables, inspired by earlier work from Maddison et al (ICLR 2017) where a similar transformation was used to learn over discrete variable prediction tasks with neural networks. Here the application is to learn sparse feedforward networks in standard classification tasks, although the framework described is quite general and could be used to impose L0 sparsity to any objective function in principal. The method provides equivalent accuracy and sparsity to published state-of-the-art results on these datasets but it is argue that learning sparsity during the training process will lead to significant speed-ups - this is demonstrated by comparing to a theoretical benchmark (standard training with dropout) rather than through empirical testing against other implementations. 
Pros:
The paper is well written and the derivation of the method is easy to follow with a good explanation of the underlying theory. 
Optimisation under L0 regularisation is a difficult and generally important topic and certainly has advantages over other sparse inference objective functions that impose shrinkage on non-sparse parameters. 
The work is put in context and related to some previous relaxation approaches to sparsity. 
The method allows for sparsity to be learned during training rather than after training (as in standard dropout approaches) and this allows the algorithm to obtain significant per-iteration speed-ups, which improves through training. 
Cons:
The method is applied to standard neural network architectures and performance in terms of accuracy and final achieved sparsity is comparable to the state-of-the-art methods. Therefore the main advance is in terms of learning speed to obtain this similar performance. However, the learning speed-up is presented against a theoretical FLOPs estimate per iteration for a similar network with dropout. It would be useful to know whether the number of iterations to achieve a particular performance is equivalent for all the different architectures considered, e.g. does the proposed sparse learning method converge at the same rate as the others? I felt a more thorough experimental section would have greatly improved the work, focussing on this learning speed aspect. 
It was unclear how much tuning of the lambda hyper-parameter, which tunes the sparsity, would be required in a practical application since tuning this parameter would increase computation time. It might be useful to provide a full Bayesian treatment so that the optimal sparsity can be chosen through hyper-parameter learning. 
Minor point: it wasn’t completely clear to me why the fact (3) is a variational approximation to a spike-and-slab is important (Appendix). I don’t see why the spike-and-slab is any more fundamental than the L0 norm prior in (2), it is just more convenient in Bayesian inference because it is an iid prior and potentially allows an informative prior over each parameter. In the context here this didn’t seem a particularly relevant addition to the paper.