SNIP: SINGLE-SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIVITY
Pruning large neural networks while maintaining their performance is often desirable due to the reduced space and time complexity. In existing methods, pruning is done within an iterative optimization procedure with either heuristically designed pruning schedules or additional hyperparameters, undermining their utility. In this work, we present a new approach that prunes a given network once at initialization prior to training. To achieve this, we introduce a saliency criterion based on connection sensitivity that identifies structurally important connections in the network for the given task. This eliminates the need for both pretraining and the complex pruning schedule while making it robust to architecture variations. After pruning, the sparse network is trained in the standard way. Our method obtains extremely sparse networks with virtually the same accuracy as the reference network on the MNIST, CIFAR-10, and Tiny-ImageNet classification tasks and is broadly applicable to various architectures including convolutional, residual and recurrent networks. Unlike existing methods, our approach enables us to demonstrate that the retained connections are indeed relevant to the given task.
Post rebuttal update/comment:
I thank the authors for the revision and have updated the score (twice!)
One genuinely perplexing result to me is that the method behaves better than random pruning, yet after selecting the salient neurons the weights can be reinitialized, as per the rebuttal:
> # Initialization procedure
- It is correct that the weights used to train the pruned model are possibly different from the ones used to compute the connection sensitivity. Given (variance scaled) initial weights, SNIP finds the architecturally important parameters in the network, then the pruned network is established and trained in the standard way.
First, there is work which states quite the opposite (e.g. https://arxiv.org/abs/1803.03635). Please relate to it.
Fundamentally, if you decouple weight pruning from initialization it also means that:
- the first layer will be pruned out of connections to constant pixels (which is seen in the visualizations), this remains meaningful even after a reinitialization
- the second and higher layers will be pruned somewhat randomly - even if the connections pruned were meaningful with the original weights, after the reinitialization the functions computed by the neurons in lower layers will be different, and have no relation to pruned weights. Thus the pruning will be essentially random (though possibly from a very specific random distribution). In other words - then neurons in a fully connected layer can be freely swapped, each neuron in the next layer behaves on al of them anyway we are thinking here about the uninitialized neurons, with each of them having a distribution over weights and not a particular set of sampled weights, this is valid because we will reinitialize the neurons). Because of that, I wouldn't call any particular weight/connection architecturally important and find it strange that such weights are found.
I find this behavior really perplexing, but I trust that your experiments are correct. however, please, if you have the time, verify it.
Original review:
The paper presents an intriguing result in which a salient, small subset of weights can be selected even in untrained networks given sensible initialization defaults are used. This result is surprising - the usual network pruning procedure assumed that a network is pretrained, and only then important connections are removed.
The contributions of the paper are two-fold:
1) it reintroduces a multiplicative sensitivity measure similar to the Breiman garotte
2) and shows which other design choices are needed to make it work on untrained networks, which is surprising.
While the main idea of the paper is clear and easy to intuitively understand, the details are not. My main concern is that paper differentiates between weights and connections (both terms are introduced on page iv to differentiate from earlier work). However, it is not clear what are the authors referring to:
- a conv layer has many repeated applications of the same weight. Am I correct to assume that a conv layer has many more connections, than weights? Furthermore, are the dramatic sparsities demonstrated over connections counted in this manner? This is important - on MNIST each digit has a constant zero border, all connections to the border are not needed and can be trivially removed (one can crop the images to remove them for similar results). Thus we can trivially remove connections, without removing weights.
- in paragraph 5.5 different weight initialization schemes are used for the purpose of saliency estimation, but the paragraph then says "Note that for training VS-X initialization is used in all the cases." Does it mean that first a set of random weights is sampled, then the sensitivities are computed, then a salient set of connections is established and the weights are REINITIALIZED from a distribution possibly different than the one used to compute the sensitivity? The fact that it works is very surprising and again suggests that the method identifies constant background pixels rather than important weights.
- on the other hand, if there is a one-to-one correspondence between connections and weights, then the differentiation from Karnin (1990) at the bottom of p. iv is misleading.
I would also be cautious about extrapolating results from MNIST to other vision datasets. MNIST has dark backgrounds. Let f(w,c) = 0*w*c. Trivially, df/dw = df/dc = 0. Thus the proposed sensitivity measure picks non-background pixels, which is also demonstrated in figure 2. However, this is a property of the dataset (which encodes background with 0) and not of the method! This should be further investigated - a quick check is to invert MNIST (make the images black-on-white, not white-on-black) and see if the method still works. Fashion MNIST behaves in a similar way. Thus the only non-trvial experiments are the ones on CIFAR10 (Table 2), but the majority of the analysis is conducted on white-on-black MNIST and Fashion-MNIST.
Finally, no experiment shows the benefit of introducing the variables "c", rather than using the gradient with respect to the weights. let f be the function computed by the network. Then:
- df/d(cw) is the gradient passed to the weights if the "c" variables were not introduced
- df/dw = df/d(cw) d(cw)/dw = df/d(cw) * c = df/d(cw)
- df/dc = df/d(cw) d(cw)/dc = df/d(cw) * w
Thus the proposed change seems to favor a combination of weight magnitude and the regular df/dw magnitude. I'd like to see how using the regular df/dw criterion would fare in single-shot pruning. In particular, I expect using the plain gradient to lead to similar selections to those in Figure 2, because for constant  pixels 0 = df/d(cw) = df/dc = df/dw.
Suggested corrections:
In related work (sec. 2) it is pointed that Hessian-based methods are unpractical due to the size od the Hessian. In fact OBD uses a diagonal approximation to the hessian, which is computed with complexity similar to the gradient, although it is typically not supported by deep learning toolkits. Please correct.
The description of weight initialization schemes should also be corrected (sec. 4.2). The sentence "Note that initializing neural networks is a random process, typically done using normal distribution with zero mean and a fixed variance." is wrong and artificially inflates the paper's contribution.  Variance normalizing schemes had been known since the nineties (see efficient backprop) and are the default in many toolkits, e.g. Pytorch uses the Kaiming rule which sets the standard deviation according to the fan-in: https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/linear.py#L56.
Please enumerate the datasets (MNIST, Fashion-MNIST, CIFAR10) in the abstract, rather than saying "vision datasets", because MNIST in particular is not representative of vision datasets due to the constant zero padding, as explained before.
Missing references:
- Efficient Backprop http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf discusses variance scaling initialization, and approximations to the hessian. Since both are mentioned in the text this should be cited as well.
- the Breiman non-negative garotte (https://www.jstor.org/stable/1269730) is a similar well-known technique in statistics
Finally, I liked the paper and wanted to give it a higher score, but reduced it because of the occurrence of many broad claims made in the paper, such as: 1) method works on MNIST => abstract claims it generally works on vision datasets 2) paper states "typically used is fixed variance init", but the popular toolkits (pytorch, keras) actually use the variance scaling one by default 3) the badly explained distinction between connection and weight and the relation that it implies to prior work. I will revise the score if these claims are corrected.