summaries: This work introduces SNIP, a simple way to prune neural network weights before training according to a specific criterion. SNIP identifies prunable weights by the normalised gradient of the loss w.r.t. an implicit multiplicative factor “c” on the weights, denoted as the “sensitivity”. Essentially, this criterion takes two factors into account when determining the relevance of each weight; the scale of the gradient and the scale of the actual weight. The authors then rank the weights according to their sensitivity and remove the ones that are not in the top-k. They then proceed to train the surviving weights as normal on the task at hand. In experiments they show that this method can offer competitive results while being much simpler to implement than other methods in the literature. The paper focuses on pruning neural networks. They propose to identify the nodes to be pruned even before training the whole network (conventionally, it is done as a separate step after the nn was trained and involves a number of iterations of retraining pruned nn). This initial step that identifies the connections to be pruned works off a mini-batch of data.
abstractText: Pruning large neural networks while maintaining their performance is often desirable due to the reduced space and time complexity. In existing methods, pruning is done within an iterative optimization procedure with either heuristically designed pruning schedules or additional hyperparameters, undermining their utility. In this work, we present a new approach that prunes a given network once at initialization prior to training. To achieve this, we introduce a saliency criterion based on connection sensitivity that identifies structurally important connections in the network for the given task. This eliminates the need for both pretraining and the complex pruning schedule while making it robust to architecture variations. After pruning, the sparse network is trained in the standard way. Our method obtains extremely sparse networks with virtually the same accuracy as the reference network on the MNIST, CIFAR-10, and Tiny-ImageNet classification tasks and is broadly applicable to various architectures including convolutional, residual and recurrent networks. Unlike existing methods, our approach enables us to demonstrate that the retained connections are indeed relevant to the given task.
authors.name: Namhoon Lee
authors.name: Thalaiyasingam Ajanthan
authors.name: Philip H. S. Torr
references.authors: Sanjeev Arora Rong Ge Behnam Neyshabur Yi Zhang
references.title: Stronger generalization bounds for deep nets via a compression approach
references.authors: Leo Breiman
references.title: Better subset regression using the nonnegative garrote
references.venue: Technometrics,
references.authors: Á. Miguel
references.title: Carreira-Perpiñán and Yerlan Idelbayev. “Learning-compression” algorithms for neural net pruning
references.authors: Yves Chauvin
references.title: A back-propagation algorithm with optimal use of hidden units
references.authors: Kyunghyun Cho Bart Van Merriënboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio
references.title: Learning phrase representations using rnn encoder-decoder for statistical machine
references.authors: Xin Dong Shangyu Chen Sinno Pan
references.title: Learning to prune deep neural networks via layer-wise optimal brain surgeon
references.authors: Xavier Glorot Yoshua Bengio
references.title: Understanding the difficulty of training deep feedforward neural networks
references.authors: Yunchao Gong Liu Liu Ming Yang Lubomir Bourdev
references.title: Compressing deep convolutional networks using vector quantization
references.venue: arXiv preprint arXiv:1412.6115,
references.authors: Yiwen Guo Anbang Yao Yurong Chen
references.title: Dynamic network surgery for efficient dnns
references.authors: Suyog Gupta Ankur Agrawal Kailash Gopalakrishnan Pritish Narayanan
references.title: Deep learning with limited numerical precision
references.authors: Song Han Jeff Pool John Tran William Dally
references.title: Learning both weights and connections for efficient neural network
references.authors: Babak Hassibi David G Stork Gregory J Wolff
references.title: Optimal brain surgeon and general network pruning
references.venue: Neural Networks,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
references.authors: Masumi Ishikawa
references.title: Structural learning with forgetting
references.venue: Neural Networks,
references.authors: Max Jaderberg Andrea Vedaldi Andrew Zisserman
references.title: Speeding up convolutional neural networks with low rank expansions
references.authors: Ehud D Karnin
references.title: A simple procedure for pruning back-propagation trained neural networks
references.venue: Neural Networks,
references.authors: Diederik P Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: ICLR,
references.authors: Pang Wei Koh Percy Liang
references.title: Understanding black-box predictions via influence functions
references.authors: Alex Krizhevsky Ilya Sutskever Geoffrey E Hinton
references.title: Imagenet classification with deep convolutional neural networks
references.authors: Quoc V Le Navdeep Jaitly Geoffrey E Hinton
references.title: A simple way to initialize recurrent networks of rectified linear units
references.authors: Yann LeCun John S Denker Sara A Solla
references.title: Optimal brain damage
references.authors: Yann LeCun Léon Bottou Genevieve B. Orr Klaus-Robert Müller
references.title: Efficient backprop
references.venue: Neural Networks: Tricks of the Trade,
references.authors: Hao Li Asim Kadav Igor Durdanovic Hanan Samet Hans Peter Graf
references.title: Pruning filters for efficient convnets
references.authors: Christos Louizos Max Welling Diederik P Kingma
references.title: Learning sparse neural networks through l0 regularization
references.venue: ICLR,
references.authors: Dmitry Molchanov Arsenii Ashukha Dmitry Vetrov
references.title: Variational dropout sparsifies deep neural networks. ICML, 2017a
references.authors: Pavlo Molchanov Stephen Tyree Tero Karras Timo Aila Jan Kautz
references.title: Pruning convolutional neural networks for resource efficient inference. ICLR, 2017b
references.authors: Michael C Mozer Paul Smolensky
references.title: Skeletonization: A technique for trimming the fat from a network via relevance assessment
references.authors: Sharan Narang Erich Elsen Gregory Diamos Shubho Sengupta
references.title: Exploring sparsity in recurrent neural networks
references.authors: Steven J Nowlan Geoffrey E Hinton
references.title: Simplifying neural networks by soft weight-sharing
references.venue: Neural Computation,
references.authors: Ameya Prabhu Girish Varma Anoop Namboodiri
references.title: Deep expander networks: Efficient deep networks from graph theory
references.authors: Russell Reed
references.title: Pruning algorithms-a survey
references.venue: Neural Networks,
references.authors: Abigail See Minh-Thang Luong Christopher D Manning
references.title: Compression of neural machine translation models via pruning
references.authors: Karen Simonyan Andrew Zisserman
references.title: Very deep convolutional networks for large-scale image recognition
references.venue: ICLR,
references.authors: Karen Ullrich Edward Meeds Max Welling
references.title: Soft weight-sharing for neural network compression
references.authors: Andreas S Weigend David E Rumelhart Bernardo A Huberman
references.title: Generalization by weightelimination with application to forecasting
references.authors: Wei Wen Chunpeng Wu Yandan Wang Yiran Chen Hai Li
references.title: Learning structured sparsity in deep neural networks
references.authors: Wojciech Zaremba Ilya Sutskever Oriol Vinyals
references.title: Recurrent neural network regularization
references.venue: arXiv preprint arXiv:1409.2329,
references.authors: Chiyuan Zhang Samy Bengio Moritz Hardt Benjamin Recht Oriol Vinyals
references.title: Understanding deep learning requires rethinking generalization
sections.heading: 1 INTRODUCTION
sections.text: Despite the success of deep neural networks in machine learning, they are often found to be highly overparametrized making them computationally expensive with excessive memory requirements. Pruning such large networks with minimal loss in performance is appealing for real-time applications, especially on resource-limited devices. In addition, compressed neural networks utilize the model capacity efficiently, and this interpretation can be used to derive better generalization bounds for neural networks (Arora et al. (2018)).
  In network pruning, given a large reference neural network, the goal is to learn a much smaller subnetwork that mimics the performance of the reference network. The majority of existing methods in the literature attempt to find a subset of weights from the pretrained reference network either based on a saliency criterion (Mozer & Smolensky (1989); LeCun et al. (1990); Han et al. (2015)) or utilizing sparsity enforcing penalties (Chauvin (1989); Carreira-Perpiñán & Idelbayev (2018)). Unfortunately, since pruning is included as a part of an iterative optimization procedure, all these methods require many expensive prune – retrain cycles and heuristic design choices with additional hyperparameters, making them non-trivial to extend to new architectures and tasks.
  In this work, we introduce a saliency criterion that identifies connections in the network that are important to the given task in a data-dependent way before training. Specifically, we discover important connections based on their influence on the loss function at a variance scaling initialization, which we call connection sensitivity. Given the desired sparsity level, redundant connections are pruned once prior to training (i.e., single-shot), and then the sparse pruned network is trained in the standard way. Our approach has several attractive properties: • Simplicity. Since the network is pruned once prior to training, there is no need for pretraining
  and complex pruning schedules. Our method has no additional hyperparameters and once pruned, training of the sparse network is performed in the standard way. • Versatility. Since our saliency criterion chooses structurally important connections, it is robust to architecture variations. Therefore our method can be applied to various architectures including convolutional, residual and recurrent networks with no modifications.
  i
  • Interpretability. Our method determines important connections with a mini-batch of data at single-shot. By varying this mini-batch used for pruning, our method enables us to verify that the retained connections are indeed essential for the given task.
  We evaluate our method on MNIST, CIFAR-10, and Tiny-ImageNet classification datasets with widely varying architectures. Despite being the simplest, our method obtains extremely sparse networks with virtually the same accuracy as the existing baselines across all tested architectures. Furthermore, we investigate the relevance of the retained connections as well as the effect of the network initialization and the dataset on the saliency score.
sections.heading: 2 RELATED WORK
sections.text: Classical methods. Essentially, early works in network pruning can be categorized into two groups (Reed (1993)): 1) those that utilize sparsity enforcing penalties; and 2) methods that prune the network based on some saliency criterion. The methods from the former category (Chauvin (1989); Weigend et al. (1991); Ishikawa (1996)) augment the loss function with some sparsity enforcing penalty terms (e.g., L0 or L1 norm), so that back-propagation effectively penalizes the magnitude of the weights during training. Then weights below a certain threshold may be removed. On the other hand, classical saliency criteria include the sensitivity of the loss with respect to the neurons (Mozer & Smolensky (1989)) or the weights (Karnin (1990)) and Hessian of the loss with respect to the weights (LeCun et al. (1990); Hassibi et al. (1993)). Since these criteria are heavily dependent on the scale of the weights and are designed to be incorporated within the learning process, these methods are prohibitively slow requiring many iterations of pruning and learning steps. Our approach identifies redundant weights from an architectural point of view and prunes them once at the beginning before training.
  Modern advances. In recent years, the increased space and time complexities as well as the risk of overfitting in deep neural networks prompted a surge of further investigation in network pruning. While Hessian based approaches employ the diagonal approximation due to its computational simplicity, impressive results (i.e., extreme sparsity without loss in accuracy) are achieved using magnitude of the weights as the criterion (Han et al. (2015)). This made them the de facto standard method for network pruning and led to various implementations (Guo et al. (2016); Carreira-Perpiñán & Idelbayev (2018)). The magnitude criterion is also extended to recurrent neural networks (Narang et al. (2017)), yet with heavily tuned hyperparameter setting. Unlike our approach, the main drawbacks of magnitude based approaches are the reliance on pretraining and the expensive prune – retrain cycles. Furthermore, since pruning and learning steps are intertwined, they often require highly heuristic design choices which make them non-trivial to be extended to new architectures and different tasks. Meanwhile, Bayesian methods are also applied to network pruning (Ullrich et al. (2017); Molchanov et al. (2017a)) where the former extends the soft weight sharing in Nowlan & Hinton (1992) to obtain a sparse and compressed network, and the latter uses variational inference to learn the dropout rate which can then be used to prune the network. Unlike the above methods, our approach is simple and easily adaptable to any given architecture or task without modifying the pruning procedure.
  Network compression in general. Apart from weight pruning, there are approaches focused on structured simplification such as pruning filters (Li et al. (2017); Molchanov et al. (2017b)), structured sparsity with regularizers (Wen et al. (2016)), low-rank approximation (Jaderberg et al. (2014)), matrix and tensor factorization (Novikov et al. (2015)), and sparsification using expander graphs (Prabhu et al. (2018)) or Erdős-Rényi random graph (Mocanu et al. (2018)). In addition, there is a large body of work on compressing the representation of weights. A non-exhaustive list includes quantization (Gong et al. (2014)), reduced precision (Gupta et al. (2015)) and binary weights (Hubara et al. (2016)). In this work, we focus on weight pruning that is free from structural constraints and amenable to further compression schemes.
sections.heading: 3 NEURAL NETWORK PRUNING
sections.text: The main hypothesis behind the neural network pruning literature is that neural networks are usually overparametrized, and comparable performance can be obtained by a much smaller network (Reed (1993)) while improving generalization (Arora et al. (2018)). To this end, the objective is to learn
  ii
  a sparse network while maintaining the accuracy of the standard reference network. Let us first formulate neural network pruning as an optimization problem.
  Given a dataset D = {(xi,yi)}ni=1, and a desired sparsity level κ (i.e., the number of non-zero weights) neural network pruning can be written as the following constrained optimization problem:
  min w L(w;D) = min w
  1
  n n∑ i=1 `(w; (xi,yi)) , (1)
  s.t. w ∈ Rm, ‖w‖0 ≤ κ . Here, `(·) is the standard loss function (e.g., cross-entropy loss), w is the set of parameters of the neural network, m is the total number of parameters and ‖ · ‖0 is the standard L0 norm. The conventional approach to optimize the above problem is by adding sparsity enforcing penalty terms (Chauvin (1989); Weigend et al. (1991); Ishikawa (1996)). Recently, Carreira-Perpiñán & Idelbayev (2018) attempts to minimize the above constrained optimization problem using the stochastic version of projected gradient descent (where the projection is accomplished by pruning). However, these methods often turn out to be inferior to saliency based methods in terms of resulting sparsity and require heavily tuned hyperparameter settings to obtain comparable results.
  On the other hand, saliency based methods treat the above problem as selectively removing redundant parameters (or connections) in the neural network. In order to do so, one has to come up with a good criterion to identify such redundant connections. Popular criteria include magnitude of the weights, i.e., weights below a certain threshold are redundant (Han et al. (2015); Guo et al. (2016)) and Hessian of the loss with respect to the weights, i.e., the higher the value of Hessian, the higher the importance of the parameters (LeCun et al. (1990); Hassibi et al. (1993)), defined as follows:
  sj =  |wj | , for magnitude based w2jHjj
  2 or w2j
  2H−1jj for Hessian based .
  (2)
  Here, for connection j, sj is the saliency score, wj is the weight, and Hjj is the value of the Hessian matrix, where the Hessian H = ∂2L/∂w2 ∈ Rm×m. Considering Hessian based methods, the Hessian matrix is neither diagonal nor positive definite in general, approximate at best, and intractable to compute for large networks.
  Despite being popular, both of these criteria depend on the scale of the weights and in turn require pretraining and are very sensitive to the architectural choices. For instance, different normalization layers affect the scale of the weights in a different way, and this would non-trivially affect the saliency score. Furthermore, pruning and the optimization steps are alternated many times throughout training, resulting in highly expensive prune – retrain cycles. Such an exorbitant requirement hinders the use of pruning methods in large-scale applications and raises questions about the credibility of the existing pruning criteria.
  In this work, we design a criterion which directly measures the connection importance in a datadependent manner. This alleviates the dependency on the weights and enables us to prune the network once at the beginning, and then the training can be performed on the sparse pruned network. Therefore, our method eliminates the need for the expensive prune – retrain cycles, and in theory, it can be an order of magnitude faster than the standard neural network training as it can be implemented using software libraries that support sparse matrix computations.
sections.heading: 4 SINGLE-SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIVITY
sections.text: Given a neural network and a dataset, our goal is to design a method that can selectively prune redundant connections for the given task in a data-dependent way even before training. To this end, we first introduce a criterion to identify important connections and then discuss its benefits.
sections.heading: 4.1 CONNECTION SENSITIVITY: ARCHITECTURAL PERSPECTIVE
sections.text: Since we intend to measure the importance (or sensitivity) of each connection independently of its weight, we introduce auxiliary indicator variables c ∈ {0, 1}m representing the connectivity of
  iii
  parameters w.1 Now, given the sparsity level κ, Equation 1 can be correspondingly modified as:
  min c,w L(c w;D) = min c,w
  1
  n n∑ i=1 `(c w; (xi,yi)) , (3)
  s.t. w ∈ Rm , c ∈ {0, 1}m, ‖c‖0 ≤ κ ,
  where denotes the Hadamard product. Compared to Equation 1, we have doubled the number of learnable parameters in the network and directly optimizing the above problem is even more difficult. However, the idea here is that since we have separated the weight of the connection (w) from whether the connection is present or not (c), we may be able to determine the importance of each connection by measuring its effect on the loss function.
  For instance, the value of cj indicates whether the connection j is active (cj = 1) in the network or pruned (cj = 0). Therefore, to measure the effect of connection j on the loss, one can try to measure the difference in loss when cj = 1 and cj = 0, keeping everything else constant. Precisely, the effect of removing connection j can be measured by,
  ∆Lj(w;D) = L(1 w;D)− L((1− ej) w;D) , (4) where ej is the indicator vector of element j (i.e., zeros everywhere except at the index j where it is one) and 1 is the vector of dimension m.
  Note that computing ∆Lj for each j ∈ {1 . . .m} is prohibitively expensive as it requires m + 1 (usually in the order of millions) forward passes over the dataset. In fact, since c is binary, L is not differentiable with respect to c, and it is easy to see that ∆Lj attempts to measure the influence of connection j on the loss function in this discrete setting. Therefore, by relaxing the binary constraint on the indicator variables c, ∆Lj can be approximated by the derivative of L with respect to cj , which we denote gj(w;D). Hence, the effect of connection j on the loss can be written as:
  ∆Lj(w;D) ≈ gj(w;D) = ∂L(c w;D)
  ∂cj
  ∣∣∣∣ c=1 = lim δ→0 L(c w;D)− L((c− δ ej) w;D) δ ∣∣∣∣ c=1 .
  (5) In fact, ∂L/∂cj is an infinitesimal version of ∆Lj , that measures the rate of change of L with respect to an infinitesimal change in cj from 1 → 1 − δ. This can be computed efficiently in one forward-backward pass using automatic differentiation, for all j at once. Notice, this formulation can be viewed as perturbing the weight wj by a multiplicative factor δ and measuring the change in loss. This approximation is similar in spirit to Koh & Liang (2017) where they try to measure the influence of a datapoint to the loss function. Here we measure the influence of connections. Furthermore, ∂L/∂cj is not to be confused with the gradient with respect to the weights (∂L/∂wj), where the change in loss is measured with respect to an additive change in weight wj .
  Notably, our interest is to discover important (or sensitive) connections in the architecture, so that we can prune unimportant ones in single-shot, disentangling the pruning process from the iterative optimization cycles. To this end, we take the magnitude of the derivatives gj as the saliency criterion. Note that if the magnitude of the derivative is high (regardless of the sign), it essentially means that the connection cj has a considerable effect on the loss (either positive or negative), and it has to be preserved to allow learning on wj . Based on this hypothesis, we define connection sensitivity as the normalized magnitude of the derivatives:
  sj = |gj(w;D)|∑m k=1 |gk(w;D)| . (6)
  Once the sensitivity is computed, only the top-κ connections are retained, where κ denotes the desired number of non-zero weights. Precisely, the indicator variables c are set as follows:
  cj = 1[sj − s̃κ ≥ 0] , ∀ j ∈ {1 . . .m} , (7) where s̃κ is the κ-th largest element in the vector s and 1[·] is the indicator function. Here, for exactly κ connections to be retained, ties can be broken arbitrarily.
  We would like to clarify that the above criterion (Equation 6) is different from the criteria used in early works by Mozer & Smolensky (1989) or Karnin (1990) which do not entirely capture the
  1Multiplicative coefficients (similar to c) were also used for subset regression in Breiman (1995).
  iv
  Algorithm 1 SNIP: Single-shot Network Pruning based on Connection Sensitivity Require: Loss function L, training dataset D, sparsity level κ . Refer Equation 3 Ensure: ‖w∗‖0 ≤ κ
  1: w← VarianceScalingInitialization . Refer Section 4.2 2: Db = {(xi,yi)}bi=1 ∼ D . Sample a mini-batch of training data 3: sj ←
  |gj(w;Db)|∑m k=1|gk(w;Db)|
  , ∀j ∈ {1 . . .m} . Connection sensitivity 4: s̃← SortDescending(s) 5: cj ← 1[sj − s̃κ ≥ 0] , ∀ j ∈ {1 . . .m} . Pruning: choose top-κ connections 6: w∗ ← arg minw∈Rm L(c w;D) . Regular training 7: w∗ ← c w∗
  connection sensitivity. The fundamental idea behind them is to identify elements (e.g. weights or neurons) that least degrade the performance when removed. This means that their saliency criteria (i.e. −∂L/∂w or −∂L/∂α; α refers to the connectivity of neurons), in fact, depend on the loss value before pruning, which in turn, require the network to be pre-trained and iterative optimization cycles to ensure minimal loss in performance. They also suffer from the same drawbacks as the magnitude and Hessian based methods as discussed in Section 3. In contrast, our saliency criterion (Equation 6) is designed to measure the sensitivity as to how much influence elements have on the loss function regardless of whether it is positive or negative. This criterion alleviates the dependency on the value of the loss, eliminating the need for pre-training. These fundamental differences enable the network to be pruned at single-shot prior to training, which we discuss further in the next section.
sections.heading: 4.2 SINGLE-SHOT PRUNING AT INITIALIZATION
sections.text: Note that the saliency measure defined in Equation 6 depends on the value of weights w used to evaluate the derivative as well as the dataset D and the loss function L. In this section, we discuss the effect of each of them and show that it can be used to prune the network in single-shot with initial weights w.
  Firstly, in order to minimize the impact of weights on the derivatives ∂L/∂cj , we need to choose these weights carefully. For instance, if the weights are too large, the activations after the non-linear function (e.g., sigmoid) will be saturated, which would result in uninformative gradients. Therefore, the weights should be within a sensible range. In particular, there is a body of work on neural network initialization (Goodfellow et al. (2016)) that ensures the gradients to be in a reasonable range, and our saliency measure can be used to prune neural networks at any such initialization.
  Furthermore, we are interested in making our saliency measure robust to architecture variations. Note that initializing neural networks is a random process, typically done using normal distribution. However, if the initial weights have a fixed variance, the signal passing through each layer no longer guarantees to have the same variance, as noted by LeCun et al. (1998). This would make the gradient and in turn our saliency measure, to be dependent on the architectural characteristics. Thus, we advocate the use of variance scaling methods (e.g., Glorot & Bengio (2010)) to initialize the weights, such that the variance remains the same throughout the network. By ensuring this, we empirically show that our saliency measure computed at initialization is robust to variations in the architecture.
  Next, since the dataset and the loss function defines the task at hand, by relying on both of them, our saliency criterion in fact discovers the connections in the network that are important to the given task. However, the practitioner needs to make a choice on whether to use the whole training set, or a mini-batch or the validation set to compute the connection saliency. Moreover, in case there are memory limitations (e.g., large model or dataset), one can accumulate the saliency measure over multiple batches or take an exponential moving average. In our experiments, we show that using only one mini-batch of a reasonable number of training examples can lead to effective pruning.
  Finally, in contrast to the previous approaches, our criterion for finding redundant connections is simple and directly based on the sensitivity of the connections. This allows us to effectively identify and prune redundant connections in a single step even before training. Then, training can be performed on the resulting pruned (sparse) network. We name our method SNIP for Single-shot Network Pruning, and the complete algorithm is given in Algorithm 1.
  v
sections.heading: 5 EXPERIMENTS
sections.text: We evaluate our method, SNIP, on MNIST, CIFAR-10 and Tiny-ImageNet classification tasks with a variety of network architectures. Our results show that SNIP yields extremely sparse models with minimal or no loss in accuracy across all tested architectures, while being much simpler than other state-of-the-art alternatives. We also provide clear evidence that our method prunes genuinely explainable connections rather than performing blind pruning.
  Experiment setup For brevity, we define the sparsity level to be κ̄ = (m − κ)/m · 100 (%), where m is the total number of parameters and κ is the desired number of non-zero weights. For a given sparsity level κ̄, the sensitivity scores are computed using a batch of 100 and 128 examples for MNIST and CIFAR experiments, respectively. After pruning, the pruned network is trained in the standard way. Specifically, we train the models using SGD with momentum of 0.9, batch size of 100 for MNIST and 128 for CIFAR experiments and the weight decay rate of 0.0005, unless stated otherwise. The initial learning rate is set to 0.1 and decayed by 0.1 at every 25k or 30k iterations for MNIST and CIFAR, respectively. Our algorithm requires no other hyperparameters or complex learning/pruning schedules as in most pruning algorithms. We spare 10% of the training data as a validation set and used only 90% for training. For CIFAR experiments, we use the standard data augmentation (i.e., random horizontal flip and translation up to 4 pixels) for both the reference and sparse models. The code can be found here: https://github.com/namhoonlee/snip-public.
sections.heading: 5.1 PRUNING LENETS WITH VARYING LEVELS OF SPARSITY
sections.text: We first test our approach on two standard networks for pruning, LeNet-300-100 and LeNet-5-Caffe. LeNet-300-100 consists of three fully-connected (fc) layers with 267k parameters and LeNet-5-Caffe consists of two convolutional (conv) layers and two fc layers with 431k parameters. We prune the LeNets for different sparsity levels κ̄ and report the performance in error on the MNIST image classification task. We run the experiment 20 times for each κ̄ by changing random seeds for dataset and network initialization. The results are reported in Figure 1.
  The pruned sparse LeNet-300-100 achieves performances similar to the reference (κ̄ = 0), only with negligible loss at κ̄ = 90. For LeNet-5-Caffe, the performance degradation is nearly invisible. Note that our saliency measure does not require the network to be pre-trained and is computed at random initialization. Despite such simplicity, our approach prunes LeNets quickly (single-shot) and effectively (minimal accuracy loss) at varying sparsity levels.
sections.heading: 5.2 COMPARISONS TO EXISTING APPROACHES
sections.text: What happens if we increase the target sparsity to an extreme level? For example, would a model with only 1% of the total parameters still be trainable and perform well? We test our approach for extreme sparsity levels (e.g., up to 99% sparsity on LeNet-5-Caffe) and compare with various pruning algorithms as follows: LWC (Han et al. (2015)), DNS (Guo et al. (2016)), LC (Carreira-Perpiñán &
  vi
  Idelbayev (2018)), SWS (Ullrich et al. (2017)), SVD (Molchanov et al. (2017a)), OBD (LeCun et al. (1990)), L-OBS (Dong et al. (2017)). The results are summarized in Table 1.
  We achieve errors that are comparable to the reference model, degrading approximately 0.7% and 0.3% while pruning 98% and 99% of the parameters in LeNet-300-100 and LeNet-5-Caffe respectively. For slightly relaxed sparsities (i.e., 95% for LeNet-300-100 and 98% for LeNet-5-Caffe), the sparse models pruned by SNIP record better performances than the dense reference network. Considering 99% sparsity, our method efficiently finds 1% of the connections even before training, that are sufficient to learn as good as the reference network. Moreover, SNIP is competitive to other methods, yet it is unparalleled in terms of algorithm simplicity.
  To be more specific, we enumerate some key points and non-trivial aspects of other algorithms and highlight the benefit of our approach. First of all, the aforementioned methods require networks to be fully trained (if not partly) before pruning. These approaches typically perform many pruning operations even if the network is well pretrained, and require additional hyperparameters (e.g., pruning frequency in Guo et al. (2016), annealing schedule in Carreira-Perpiñán & Idelbayev (2018)). Some methods augment the training objective to handle pruning together with training, increasing the complexity of the algorithm (e.g., augmented Lagrangian in Carreira-Perpiñán & Idelbayev (2018), variational inference in Molchanov et al. (2017a)). Furthermore, there are approaches designed to include architecture dependent constraints (e.g., layer-wise pruning schemes in Dong et al. (2017)).
  Compared to the above approaches, ours seems to cost almost nothing; it requires no pretraining or additional hyperparameters, and is applied only once at initialization. This means that one can easily plug-in SNIP as a preprocessor before training neural networks. Since SNIP prunes the network at the beginning, we could potentially expedite the training phase by training only the survived parameters (e.g., reduced expected FLOPs in Louizos et al. (2018)). Notice that this is not possible for the aforementioned approaches as they obtain the maximum sparsity at the end of the process.
sections.heading: 5.3 VARIOUS MODERN ARCHITECTURES
sections.text: In this section we show that our approach is generally applicable to more complex modern network architectures including deep convolutional, residual and recurrent ones. Specifically, our method is applied to the following models:
  • AlexNet-s and AlexNet-b: Models similar to Krizhevsky et al. (2012) in terms of the number of layers and size of kernels. We set the size of fc layers to 512 (AlexNet-s) and to 1024 (AlexNet-b) to adapt for CIFAR-10 and use strides of 2 for all conv layers instead of using pooling layers. • VGG-C, VGG-D and VGG-like: Models similar to the original VGG models described in Simonyan & Zisserman (2015). VGG-like (Zagoruyko (2015)) is a popular variant adapted for CIFAR-10 which has one less fc layers. For all VGG models, we set the size of fc layers to 512, remove dropout layers to avoid any effect on sparsification and use batch normalization instead. • WRN-16-8, WRN-16-10 and WRN-22-8: Same models as in Zagoruyko & Komodakis (2016).
  vii
  • LSTM-s, LSTM-b, GRU-s, GRU-b: One layer RNN networks with either LSTM (Zaremba et al. (2014)) or GRU (Cho et al. (2014)) cells. We develop two unit sizes for each cell type, 128 and 256 for {·}-s and {·}-b, respectively. The model is adapted for the sequential MNIST classification task, similar to Le et al. (2015). Instead of processing pixel-by-pixel, however, we perform rowby-row processing (i.e., the RNN cell receives each row at a time).
  The results are summarized in Table 2. Overall, our approach prunes a substantial amount of parameters in a variety of network models with minimal or no loss in accuracy (< 1%). Our pruning procedure does not need to be modified for specific architectural variations (e.g., recurrent connections), indicating that it is indeed versatile and scalable. Note that prior art that use a saliency criterion based on the weights (i.e., magnitude or Hessian based) would require considerable adjustments in their pruning schedules as per changes in the model.
  We note of a few challenges in directly comparing against others: different network specifications, learning policies, datasets and tasks. Nonetheless, we provide a few comparison points that we found in the literature. On CIFAR-10, SVD prunes 97.9% of the connections in VGG-like with no loss in accuracy (ours: 97% sparsity) while SWS obtained 93.4% sparsity on WRN-16-4 but with a non-negligible loss in accuracy of 2%. There are a couple of works attempting to prune RNNs (e.g., GRU in Narang et al. (2017) and LSTM in See et al. (2016)). Even though these methods are specifically designed for RNNs, none of them are able to obtain extreme sparsity without substantial loss in accuracy reflecting the challenges of pruning RNNs. To the best of our knowledge, we are the first to demonstrate on convolutional, residual and recurrent networks for extreme sparsities without requiring additional hyperparameters or modifying the pruning procedure.
sections.heading: 5.4 UNDERSTANDING WHICH CONNECTIONS ARE BEING PRUNED
sections.text: So far we have shown that our approach can prune a variety of deep neural network architectures for extreme sparsities without losing much on accuracy. However, it is not clear yet which connections are actually being pruned away or whether we are pruning the right (i.e., unimportant) ones. What if we could actually peep through our approach into this inspection?
  Consider the first layer in LeNet-300-100 parameterized by wl=1 ∈ R784×300. This is a layer fully connected to the input where input images are of size 28 × 28 = 784. In order to understand which connections are retained, we can visualize the binary connectivity mask for this layer cl=1, by averaging across columns and then reshaping the vector into 2D matrix (i.e., cl=1 ∈ {0, 1}784×300 → R784 → R28×28). Recall that our method computes c using a minibatch of examples. In this experiment, we curate the mini-batch of examples of the same class and see which weights are retained for that mini-batch of data. We repeat this experiment for all classes
  viii
  (i.e., digits for MNIST and fashion items for Fashion-MNIST) with varying sparsity levels κ̄. The results are displayed in Figure 2 (see Appendix A for more results).
  The results are significant; important connections seem to reconstruct either the complete image (MNIST) or silhouettes (Fashion-MNIST) of input class. When we use a batch of examples of the digit 0 (i.e., the first column of MNIST results), for example, the parameters connected to the foreground of the digit 0 survive from pruning while the majority of background is removed. Also, one can easily determine the identity of items from Fashion-MNIST results. This clearly indicates that our method indeed prunes the unimportant connections in performing the classification task, receiving signals only from the most discriminative part of the input. This stands in stark contrast to other pruning methods from which carrying out such inspection is not straightforward.
sections.heading: 5.5 EFFECTS OF DATA AND WEIGHT INITIALIZATION
sections.text: Recall that our connection saliency measure depends on the network weights w as well as the given data D (Section 4.2). We study the effect of each of these in this section. Effect of data. Our connection saliency measure depends on a mini-batch of train examples Db (see Algorithm 1). To study the effect of data, we vary the batch size used to compute the saliency (|Db|) and check which connections are being pruned as well as how much performance change this results in on the corresponding sparse network. We test with LeNet-300-100 to visualize the remaining parameters, and set the sparsity level κ̄ = 90. Note that the batch size used for training remains the same as 100 for all cases. The results are displayed in Figure 3.
  Effect of initialization. Our approach prunes a network at a stochastic initialization as discussed. We study the effect of the following initialization methods: 1) RN (random normal), 2) TN (truncated random normal), 3) VS-X (a variance scaling method using Glorot & Bengio (2010)), and 4) VS-H (a variance scaling method He et al. (2015)). We test on LeNets and RNNs on MNIST and run 20 sets of experiments by varying the seed for initialization. We set the sparsity level κ̄ = 90, and train with Adam optimizer (Kingma & Ba (2015)) with learning rate of 0.001 without weight decay. Note that for training VS-X initialization is used in all the cases. The results are reported in Figure 3.
  For all models, VS-H achieves the best performance. The differences between initializers are marginal on LeNets, however, variance scaling methods indeed turns out to be essential for complex RNN models. This effect is significant especially for GRU where without variance scaling initialization, the pruned networks are unable to achieve good accuracies, even with different optimizers. Overall, initializing with a variance scaling method seems crucial to making our saliency measure reliable and model-agnostic.
  ix
  5.6 FITTING RANDOM LABELS
  To further explore the use cases of SNIP, we run the experiment introduced in Zhang et al. (2017) and check whether the sparse network obtained by SNIP memorizes the dataset. Specifically, we train LeNet-5-Caffe for both the reference model and pruned model (with κ̄ = 99) on MNIST with either true or randomly shuffled labels. To compute the connection sensitivity, always true labels are used. The results are plotted in Figure 4.
  Given true labels, both the reference (red) and pruned (blue) models quickly reach to almost zero training loss. However, the reference model provided with random labels (green) also reaches to very low training loss, even
  with an explicit L2 regularizer (purple), indicating that neural networks have enough capacity to memorize completely random data. In contrast, the model pruned by SNIP (orange) fails to fit the random labels (high training error). The potential explanation is that the pruned network does not have sufficient capacity to fit the random labels, but it is able to classify MNIST with true labels, reinforcing the significance of our saliency criterion. It is possible that a similar experiment can be done with other pruning methods (Molchanov et al. (2017a)), however, being simple, SNIP enables such exploration much easier. We provide a further analysis on the effect of varying κ̄ in Appendix B.
sections.heading: 6 DISCUSSION AND FUTURE WORK
sections.text: In this work, we have presented a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications. While SNIP results in extremely sparse models, we find that our connection sensitivity measure itself is noteworthy in that it diagnoses important connections in the network from a purely untrained network. We believe that this opens up new possibilities beyond pruning in the topics of understanding of neural network architectures, multi-task transfer learning and structural regularization, to name a few. In addition to these potential directions, we intend to explore the generalization capabilities of sparse networks.
  x
sections.heading: ACKNOWLEDGEMENTS
sections.text: This work was supported by the Korean Government Graduate Scholarship, the ERC grant ERC2012-AdG 321162-HELIOS, EPSRC grant Seebibyte EP/M013774/1 and EPSRC/MURI grant EP/N019474/1. We would also like to acknowledge the Royal Academy of Engineering and FiveAI.
sections.heading: B FITTING RANDOM LABELS: VARYING SPARSITY LEVELS
sections.heading: C TINY-IMAGENET
sections.text: 2https://tiny-imagenet.herokuapp.com/
  xiv
sections.heading: D ARCHITECTURE DETAILS
sections.text: xv
title: SNIP: SINGLE-SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIVITY