summaries: The paper proposes a multi-layer pruning method called MLPrune for neural networks, which can automatically decide appropriate compression ratios for all the layers. It firstly pre-trains a network. Then it utilizes K-FAC to approximate the Fisher matrix, which in turn approximates the exact Hessian matrix of training loss w.r.t model weights. The approximated Hessian matrix is then used to estimate the increment of loss after pruning a connection. The connections from all layers with the smallest loss increments are pruned and the network is re-trained to the final model. This paper introduces an approach to pruning the parameters of a trained neural network. The idea is inspired by the Optimal Brain Surgeon method, which relies on second derivatives of the loss w.r.t. the network parameters. Here, the corresponding Hessian matrix is approximated using the Fisher information to make the algorithm scalable to very deep networks.
abstractText: Model compression can significantly reduce the computation and memory footprint of large neural networks. To achieve a good trade-off between model size and accuracy, popular compression techniques usually rely on hand-crafted heuristics and require manually setting the compression ratio of each layer. This process is typically costly and suboptimal. In this paper, we propose a Multi-Layer Pruning method (MLPrune), which is theoretically sound, and can automatically decide appropriate compression ratios for all layers. Towards this goal, we use an efficient approximation of the Hessian as our pruning criterion, based on a Kroneckerfactored Approximate Curvature method. We demonstrate the effectiveness of our method on several datasets and architectures, outperforming previous state-of-theart by a large margin. Our experiments show that we can compress AlexNet and VGG16 by 25x without loss in accuracy on ImageNet. Furthermore, our method has much fewer hyper-parameters and requires no expert knowledge.
references.authors: Jimmy Ba Roger Grosse James Martens
references.title: Distributed second-order optimization using kroneckerfactored approximations
references.authors: Matthieu Courbariaux Itay Hubara Daniel Soudry Ran El-Yaniv Yoshua Bengio
references.title: Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1
references.venue: arXiv preprint arXiv:1602.02830,
references.authors: Emily L Denton Wojciech Zaremba Joan Bruna Yann LeCun Rob Fergus
references.title: Exploiting linear structure within convolutional networks for efficient evaluation
references.venue: In Advances in neural information processing systems,
references.authors: Xin Dong Shangyu Chen Sinno Pan
references.title: Learning to prune deep neural networks via layer-wise optimal brain surgeon
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Yunchao Gong Liu Liu Ming Yang Lubomir Bourdev
references.title: Compressing deep convolutional networks using vector quantization
references.venue: arXiv preprint arXiv:1412.6115,
references.authors: Roger Grosse James Martens
references.title: A kronecker-factored approximate fisher matrix for convolution layers
references.venue: In International Conference on Machine Learning,
references.authors: Yiwen Guo Anbang Yao Yurong Chen
references.title: Dynamic network surgery for efficient dnns
references.venue: In Advances In Neural Information Processing Systems,
references.authors: Song Han Huizi Mao William J Dally
references.title: Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding
references.venue: arXiv preprint arXiv:1510.00149,
references.authors: Song Han Jeff Pool John Tran William Dally
references.title: Learning both weights and connections for efficient neural network
references.venue: In Advances in neural information processing systems,
references.authors: Babak Hassibi David G Stork
references.title: Second order derivatives for network pruning: Optimal brain surgeon
references.venue: In Advances in neural information processing systems,
references.authors: Max Jaderberg Andrea Vedaldi Andrew Zisserman
references.title: Speeding up convolutional neural networks with low rank expansions
references.venue: arXiv preprint arXiv:1405.3866,
references.authors: Alex Krizhevsky Geoffrey Hinton
references.title: Learning multiple layers of features from tiny images
references.authors: Alex Krizhevsky Ilya Sutskever Geoffrey E Hinton
references.title: Imagenet classification with deep convolutional neural networks. In Advances in neural information processing
references.authors: Vadim Lebedev Yaroslav Ganin Maksim Rakhuba Ivan Oseledets Victor Lempitsky
references.title: Speeding-up convolutional neural networks using fine-tuned cp-decomposition
references.venue: arXiv preprint arXiv:1412.6553,
references.authors: Yann LeCun John S Denker Sara A Solla
references.title: Optimal brain damage
references.venue: In Advances in neural information processing systems,
references.authors: Yann LeCun Léon Bottou Yoshua Bengio Patrick Haffner
references.title: Gradient-based learning applied to document recognition
references.venue: Proceedings of the IEEE,
references.authors: Hao Li Asim Kadav Igor Durdanovic Hanan Samet Hans Peter Graf
references.title: Pruning filters for efficient convnets
references.venue: arXiv preprint arXiv:1608.08710,
references.authors: James Martens Roger Grosse
references.title: Optimizing neural networks with kronecker-factored approximate curvature
references.venue: In International conference on machine learning,
references.authors: Alexander Novikov Dmitrii Podoprikhin Anton Osokin Dmitry P Vetrov
references.title: Tensorizing neural networks
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Mohammad Rastegari Vicente Ordonez Joseph Redmon Ali Farhadi
references.title: Xnor-net: Imagenet classification using binary convolutional neural networks
references.venue: In European Conference on Computer Vision,
references.authors: Jorma Rissanen
references.title: Modeling by shortest data
references.venue: description. Automatica,
references.authors: Karen Simonyan Andrew Zisserman
references.title: Very deep convolutional networks for large-scale image recognition
references.venue: arXiv preprint arXiv:1409.1556,
references.authors: Wei Wen Chunpeng Wu Yandan Wang Yiran Chen Hai Li
references.title: Learning structured sparsity in deep neural networks
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Jiaxiang Wu Cong Leng Yuhang Wang Qinghao Hu Jian Cheng
references.title: Quantized convolutional neural networks for mobile devices
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Yuhuai Wu Elman Mansimov Roger B Grosse Shun Liao Jimmy Ba
references.title: Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation
references.venue: In Advances in neural information processing systems,
references.authors: Guodong Zhang Shengyang Sun David Duvenaud Roger Grosse
references.title: Noisy natural gradient as variational inference
references.venue: arXiv preprint arXiv:1712.02390,
references.authors: Chenzhuo Zhu Song Han Huizi Mao William J Dally
references.title: Trained ternary quantization
references.venue: arXiv preprint arXiv:1612.01064,
sections.heading: 1 INTRODUCTION
sections.text: Deep neural networks have proven very successful in many artificial intelligence tasks such as computer vision, natural language processing, and robotic control. In exchange for such success, modern architectures are usually composed of many stacked layers parameterized with a large number of learnable weights. This contrasts classical networks which have hundreds or even thousands times fewer parameters. As a consequence, modern architectures require considerable memory storage and intensive computation. This is problematic in applications that need to run on small embedded systems or that require low-latency to make safety-critical decisions.
  Fortunately, we can compress these large networks with little to no loss of accuracy, by exploiting the fact that many redundancies exist within their parameters. A popular approach is to quantize the parameters using lower precision, thus encoding the network with fewer bits (Courbariaux et al., 2016; Gong et al., 2014; Rastegari et al., 2016; Wu et al., 2016; Zhu et al., 2016). On the other hand, pruning techniques (Dong et al., 2017; Guo et al., 2016; Han et al., 2015a;b; Li et al., 2016; Wen et al., 2016) aim at removing redundant connections. It has been widely exploited due to its simplicity and efficacy. In this paper, we follow this line of work and propose a new pruning approach.
  Pruning algorithms differ on the pruning criteria employed. Popular pruning methods typically rely on heuristics, such as weight magnitudes. However, a small magnitude does not necessarily mean unimportance (Hassibi & Stork, 1993; LeCun et al., 1990); if the input neuron has a large expected value, a small weight could still have a large effect on its output neuron. As a consequence, magnitudebased pruning might delete important parameters, or preserve unimportant ones. Furthermore, magnitude-based pruning requires manually setting the compression ratio of each layer. These ratios are not easy to tune, as different layers (and architectures) typically have different sensitivities for compression. Directly comparing the parameters’ magnitudes from all layers and pruning the smallest one would not solve the problem, since magnitudes are not calibrated across layers. Because of this, existing approaches usually involve time-consuming trial and error processes that require domain expertise.
  In this paper we employ the Hessian matrix as a principled pruning criterion, as it characterizes the local curvature of the training loss. The Hessian was exploited in the past, but was either assumed to be computable (Hassibi & Stork, 1993), which is not possible in practice for modern neural networks with millions of parameters, or simplistic approximations (e.g., diagonal (LeCun et al., 1990)) were
  utilized resulting in poor performance. Here we first note that the Fisher Information matrix is close to the Hessian under certain conditions. We then use a Kronecker-factored Approximate Curvature (K-FAC) (Martens & Grosse, 2015) method to efficiently estimate the Fisher matrix. In our experiments, the overhead of estimating the Hessian approximation is very small compared with the pre-train and re-train time. Therefore, our method is efficient to use, while exhibiting better compression performance compared with previous second-order pruning. Importantly, this criterion is also calibrated across layers, allowing us to directly compare parameters from different layers and set one global compression ratio for the whole network, rather than one ratio per layer. This makes our method significantly easier to use when compared to magnitude-based pruning. Our method achieves state-of-the-art results on several benchmark datasets and model architectures. Furthermore, it can be easily applied to new customized network architectures, without costly tuning many compression hyper-parameters.
sections.heading: 2 BACKGROUND
sections.text: In this section, we first introduce the notation used in the paper. Since our work is inspired by the Optimal Brain Surgeon (Hassibi & Stork, 1993), we also briefly review the idea and its limitations.
sections.heading: 2.1 NOTATION
sections.text: Suppose we have a neural network with L fully-connected layers1. Let p(y|x,Θ) be the output distribution and L be the loss function. y denotes the target outputs, x the inputs, and Θ the network’s weights. Each layer l is associated with its input al−1 ∈ Rdl−1 , a weight matrix Wl ∈ Rdl−1×dl , a binary mask matrix Γl ∈ Rdl−1×dl and its output sl ∈ Rdl . The elements γ(i,j)l ∈ {0, 1} in the binary matrix Γl indicate whether the corresponding weight is retained or pruned away. We use subscript to denote the layer index, and superscript to denote the element index in a matrix. We also distinguish a matrix (vector) from a scalar by capital boldface2. The weights in a layer l can be computed as θ(i,j)l = w (i,j) l γ (i,j) l , and thus the forward-pass can be written as,
  sl = (Wl Γl) al−1, al = h (sl) , (1)
  where denotes element-wise product, and h is the Relu activation in our experiments. We also use W to denote the concatenation of [vec{W1}, vec{W2}, · · · , vec{WL}], which is a column vector with the same size as the total number of parameters in this neural network, similarly for Γ and Θ. Index q denotes the index number of a parameter in W(or Γ, Θ) that is associated with w(i,j)l (or γ (i,j) l , θ (i,j) l ). Parameters, weights and connections are used interchangeably.
sections.heading: 2.2 A REVIEW ON OPTIMAL BRAIN SURGEON (OBS)
sections.text: Given a neural network with parameters Θ, the local surface of the training loss can be characterized by its Taylor expansion:
  δL = ∂L ∂Θ δΘ + 1 2 δΘTHδΘ +O(||δΘ||3), (2)
  where δΘ is a small change of the parameter values (e.g., setting some parameters to zero) and H is the Hessian matrix defined as ∂2L/∂Θ2. Note that at convergence the first-order term vanishes and the higher-order term can be neglected. OBS aims to find an index q, such that when changing θq from its original value to zero, the change in training loss δL is minimized:
  min q ( min δΘ ( 1 2 δΘTHδΘ )) , s.t. eTq δΘ + δΘ = 0, (3)
  where eTq is the unit vector in the weight space corresponding to θq . The inner minimization problem can be solved by the method of Lagrangian multipliers. Hassibi & Stork (1993) applied OBS on a small network with 18,000 weights and showed good results. However, since the exact solution involves the inverse of the Hessian matrix, original OBS is intractable for modern neural networks that contain millions of parameters.
  1Convolutional layers can be transformed into fully-connected layers if we vectorize filters within each channel and expand the input feature map with repeated patches
  2with minor abuse of notation for al−1 and sl
sections.heading: 3 MULTI-LAYER PRUNING
sections.text: Our method follows a similar pipeline when compared to conventional pruning methods. We first pre-train a neural network until converged, and then apply Multi-Layer Pruning (including surgeon) upon it. This will give us a compressed network with slightly worse performance. Finally, the network is re-trained to recover the model performance. In the following paragraph, we first introduce the problem formulation and its ideal solution in Section 3.1. We then show how to make approximations in Section 3.2.
sections.heading: 3.1 MULTI-LAYER PRUNING AND SURGEON
sections.text: Our objective is to prune as much as possible while maintaining good model performance, e.g. high accuracy in classification. This is similar to the objective of Minimum Description Length (MDL, (Rissanen, 1978))
  min Θ D(data|Θ) +D(Θ). (4)
  In our case, the first term in Eq. (4) is equivalent to the training loss, and the second term is the description length denoting how many bits we need to encode the model. Here, we use the number of parameters as an indicator of such description length. We also introduce a relative importance λ, which characterizes the trade-off between sparsity and model performance, and controls the final size of a compressed model. Our objective can then be written as
  Ψ = min W,Γ L (y|x,W Γ) + λ L∑ l=1 ∑ (i,j) γ (i,j) l  , (5) where Γ is the binary mask matrix defined in Section 2.1. This objective cannot be directly optimized with SGD as γ(i,j)l is contrained to be binary. Instead, we consider an easier case where we first pre-train the model to the local minimum of L, then update one γ(i,j)l from 1 to 0 if such change will decrease Ψ, or equivalently, if such change will not increase L more than λ. Since the parameter for forward-pass is θ(i,j)l = γ (i,j) l w (i,j) l , updating γ (i,j) l is equivalent to prune a parameter θ (i,j) l . Similar to Hassibi & Stork (1993), we also consider a surgeon operation after pruning. This will change other preserved parameters accordingly, so as to minimize the loss increment caused by the pruning. More specifically, after pruning one parameter θ(i,j)l and applying the surgeon, the increment of L will be,
  min δΘ
  ( 1
  2 δΘTHδΘ
  ) , s.t. δθ
  (i,j) l + w (i,j) l = 0. (6)
  δθ (i,j) l is the corresponding scalar value in δΘ, and H is the Hessian matrix of L w.r.t (W Γ). Solving such a minimization problem with Lagrangian multipliers results in the following update
  ∆Lq = 1
  2
  ( w
  (i,j) l )2 [H−1] (q,q) , δΘ∗ = [ − w (i,j) l [H−1] (q,q) H−1 ](.,q) . (7)
  Here, δΘ∗ is the surgeon applied on the remaining Θ (and W accordingly). ∆Lq is the increment of loss after pruning and the surgeon operation. Again, superscript (q, q) denotes the element at the qth row and qth column, (., q) denotes the qth column in that matrix, and q is the index of the parameter in H that associates with w(i,j)l . The estimation of H −1 will be introduced in section 3.2.
  Theoretically, we can calculate ∆Lq for each parameter and prune the one with the smallest ∆L, but this will become impractically time-consuming. Therefore, at each pruning step, we calculate ∆Lq for each remaining parameter, and prune all parameters with ∆Lq < λ. This is fine as long as such assumption holds, that H does not change significantly when we prune those unimportant parameters one by one. In our experiments, we simply set λ to be the pth percentile of all ∆Lq from all layers. Therefore, our method can jointly determine the importance of parameters from all layers and prune multiple layers simultaneously.
  After pruning and doing surgeon, we apply SGD to those retained parameters to re-train the network,
  W←W − α∂L (W Γ) ∂ (W Γ) Γ. (8)
sections.heading: 3.2 APPROXIMATING HESSIAN USING FISHER
sections.text: Performing the pruning and surgeon operations as in Eq. (7) involves estimating and inverting the Hessian matrix, which is intractable for modern neural networks that contain millions of parameters. To efficiently calculate Eq. (7), we approximate the Hessian. Towards this goal, we first employ the Fisher Information matrix to approximate the Hessian and further use a Kronecker-factored Approximate Curvature (K-FAC) method to approximate the Fisher matrix. The first approximation comes from the fact that if the training objective is the negative log-likelihood, the Hessian matrix consists of expected second-order derivatives under the data distribution, while the Fisher matrix consists of expected second-order derivatives under the model distribution. Since modern neural networks usually have strong model capacities, we expect those two distributions are close for a well-trained model. The second approximation is demonstrated to be reasonable in optimization tasks (Grosse & Martens, 2016; Martens & Grosse, 2015), and will further help us to calculate Eq. (7) efficiently.
  The K-FAC method approximates the Fisher matrix as a block-diagonal matrix and estimates each block by the Kronecker-product of two much smaller matrices, which are the second-order statistics of inputs and derivatives. Given a neural network with stacked fully-connected layers3, the Fisher Information matrix is defined as
  F = E [ (∇WL) (∇WL)T ] . (9)
  Unless specified otherwise, the expectation is taken with respect to the model distribution. We can re-write the Fisher in a block-wise manner by partitioning rows and columns of F if they correspond to parameters within the same layer. The (i, j) block is then4
  Fij = E [ vec {∇WiL} vec { ∇WjL }T ] , (10)
  which is the covariance of the gradients from i and j layers5. As noted by Martens & Grosse (2015), the elements of the covariance of derivatives from different layers are generally smaller than the ones within the same layer. Therefore, F can be approximated by a block-diagonal matrix where the lth block is simply Fll.
  Using back-propagation, the gradients of layer l can be calculated as∇WlL = (∇slL) (al−1) T . The lth diagonal block Fll can then be written as
  Fll = E [ vec { (∇slL) (al−1) T } vec { (∇slL) (al−1) T }T] = E [ al−1a T l−1 ⊗ (∇slL)(∇slL)T ] (11)
  ≈ E [ al−1a T l−1 ] ⊗ E [ (∇slL)(∇slL)T ] , (12)
  where ⊗ denotes Kronecker-product. The second equality comes from the property of the Kroneckerproduct6. The last approximation is supposed to maintain the "coarse structure" of the Fisher matrix (Martens & Grosse, 2015), and such "coarse structure" is sufficient for our method to achieve a good compression ratio as empirically demonstrated in our experiments. Therefore, F (and thus H) can be approximated by several much smaller matrices
  Al−1 = E [ al−1a T l−1 ] , DSl = E [ (∇slL)(∇slL)T ] , Fll = Al−1 ⊗DSl. (13)
  For a typical modern neural network such as AlexNet, the original Hessian matrix is a 61M× 61M matrix, while Al−1 and DSl of the largest fully-connected layer have sizes of only 9126 × 9126 and 4096 × 4096 respectively. We use exponential moving average to estimate the expectation in Al−1 and DSl, with a decay factor of 0.95 and a horizon of 1000 steps in all experiments. This adds only a small overhead to the normal forward-backward pass. Furthermore, the inverse F−1 and matrix-vector product F−1h can be efficiently calculated by leveraging the block diagnoal structure and property of Kronecker-product7.
  3Details for convolutional layers are discussed in Grosse & Martens (2016). 4Note that vec{} denotes vectorization, as Wi is a matrix, while previously W is a vector. See section2.1 5E [∇WL] = 0, since the expectation is taken over the model distribution 6vec { uvT } = v ⊗ u, (A⊗B)(C⊗D) = (A⊗C)(B⊗D). 7(A⊗B)−1 = A−1 ⊗B−1, (A⊗B)vec{X} = vec{BXAT }
  Algorithm 1 Multi-Layer Pruning Initialization: W, Γ, Pruning fraction p. Pre-training Stage:
  1: Pre-train the network Pruning Stage:
  2: for t = 0, · · · , 1000 do 3: Update Al−1 ← 0.95×Al−1 + 0.05× al−1aTl−1 4: Update DSl ← 0.95×DSl + 0.05× (∇slL)(∇slL)T 5: end for 6: Compute Fisher matrix F by Eq. (13). 7: Compute importance measure ∆Lq for each parameter by Eq. (7). 8: Normalize ∆Lq within each layer by Eq. (14). 9: Compute pth percentile of ˜∆Lq from all layers as λ.
  10: Update mask γ(i,j)l to 0 if its corresponding ˜∆Lq is smaller than λ. 11: Compute δΘ∗ by Eq. (7) and update Θ,W← Θ + δΘ∗. 12: Re-train the network by Eq. (8).
  However, such a block diagonal approximation decorrelates the gradients from different layers, and might lead to different scales for different layers. This will become problematic when we jointly prune all layers, i.e. sorting ∆Lq from all layers. We empirically find that ∆Lq from different layers will sometimes differ by several orders of magnitudes, which makes it impossible to perform the multi-layer pruning. Therefore, we propose a simple yet effective modification for this problem. After calculating ∆Lq , we normalize it within the same layer:
  ˜∆Lq = ∆Lq∑
  q∈layer l ∆Lq (14)
  Then, we use ˜∆Lq as an importance measure to prune parameters. Surgeon and other parts remain unchanged. We show our algorithm as Algorithm 1. During pruning, we only need to set an overall compression ratio, and the MLPrune method will automatically decide the compression ratio for each layer. This free us from tuning hyper-parameters, and thus it is easier and more efficient to apply.
sections.heading: 4 RELATED WORK
sections.text: Model compression task aims to compress a large model into a smaller one while maintaining good performance. There are several popular families of approaches, including pruning, quantization, and low-rank approximation. Quantization (Courbariaux et al., 2016; Gong et al., 2014; Rastegari et al., 2016; Wu et al., 2016; Zhu et al., 2016) aims to use fewer bits to encode each parameter, e.g. binary neural network. Low-rank approximation (Denton et al., 2014; Jaderberg et al., 2014; Lebedev et al., 2014; Novikov et al., 2015) approximates layer weights by low-rank representations, and thus saves the storage and makes speedup. Pruning, being one of the most popular methods due to its simplicity and effectiveness, aims to delete unimportant parameters from a large network. These techniques could be further integrated together and result in better compression ratio (Han et al., 2015a).
  The pruning operation involves finding and deleting unimportant parameters, and thus needs a criterion for deciding importance. Magnitude-based methods (Guo et al., 2016; Han et al., 2015b; Li et al., 2016; Wen et al., 2016) use the absolute value of a parameter as an indicator, assuming smaller weights generally have smaller importance upon a network’s output, but there is no theoretical evidence to support this criterion. On the other hand, some other work (Hassibi & Stork, 1993; LeCun et al., 1990) use the second-order Hessian matrix to calculate the importance of each parameter, and show significant improvements when applied to toy neural networks. However, these methods cannot be easily adapted to modern neural networks due to the large size of the Hessian matrix. Recently, Dong et al. (2017) use the layer-wise reconstruction as an indicator, which provides a theoretical upper bound for the pruning error, while avoiding the intractable large Hessian matrix. However, all these methods require specifying one compression ratio for each layer, and searching such ratios are time-consuming for very deep neural networks.
  In our work, we utilize K-FAC to approximate the Fisher matrix, which in turn approximates the exact Hessian matrix. K-FAC(Grosse & Martens, 2016; Martens & Grosse, 2015) provides an efficient way to estimate and invert an approximation of the Fisher matrix of a neural network. It first approximates the Fisher by a block diagonal matrix, and then decomposes each block by two much smaller matrix via Kronecker-product. Such approximation and its variants(Ba et al., 2016; Wu et al., 2017; Zhang et al., 2017) have shown success in the field of optimization.
sections.heading: 5 EXPERIMENTS
sections.text: Following previous work, we use compression ratio as our evaluation metric, which is defined as the ratio of the number of parameters that remaining active after pruning to the number of original parameters. To show the generality of our method, we perform experiments using three different datasets and five different deep network architectures, from shallow to deep. This includes LeNet300-100 and LeNet-5 (LeCun et al., 1998) on MNIST, CifarNet (Krizhevsky & Hinton, 2009) on Cifar-10, and AlexNet (Krizhevsky et al., 2012) and VGG16(Simonyan & Zisserman, 2014) on Imagenet ILSVRC-2012. The aforementioned architectures contain both fully-connected layers and convolution layers, and have different sizes from 267K to 138M parameters. We compare our results with some strong pruning baselines, including: 1) Randomly Pruning (Dong et al., 2017), 2) OBD (LeCun et al., 1990), 3) LWC (Han et al., 2015b), 4) DNS (Guo et al., 2016), 5) L-OBS (Dong et al., 2017). Following these work, we also apply pruning in an iterative manner and report the results in Table 1. Due to minor differences in implementation, different papers report different performance of pre-trained models. Therefore, we also report the original error and ∆error before and after pruning for each method. More implementation details could be found in Appendix A.
sections.heading: 5.1 DOES MLPRUNE ACHIEVES BETTER COMPRESSION RESULTS?
sections.text: MNIST: We first conduct experiments on the MNIST dataset with LeNet-300-100 and LeNet-5. LeNet-300-100 has 2 fully-connected layers, with 300 and 100 hidden units respectively. LeNet-5 is a CNN with 2 convolutional layers, followed by 2 fully-connected layers. Table 1 shows that we can compress LeNet-300-100 to 1.3% of original size with almost no loss in performance. Similarly for LeNet-5, we can compress to 0.5%, which is much smaller than previous best result.
  8Original DNS paper doesn’t provide result on VGG16. Therefore we adopt numbers from the reimplementation in Dong et al. (2017)
  Cifar-10: We conduct experiments on Cifar-10 image classification benchmark with Cifar-Net architecture. Cifar-Net is a variant of AlexNet, containing 3 convolutional layers and 2 fullyconnected layers. Following previous work (Dong et al., 2017), we first pre-train the network to achieve 18.43% error rate on the testing set. We then iteratively prune the network to [50%, 25%, 12.5%, 10%, 8%, 6.4%] of its original size. This gives us a better compression ratio than our baselines while maintaining similar performance.
  ImageNet: To demonstrate our method’s effectiveness on larger models and datasets, we prune AlexNet on ImageNet ILSVRC-2012. AlexNet has 5 convolutional layers, 3 fully-connected layers, and 61M parameters in total. We first pre-train the network, and achieve similar top-1 accuracy with other implementations. We then iteratively prune the network to [50%, 25%, 12.5%, 6.25%, 5%, 4%] of its original size. To the best of our knowledge, this is the best pruning result so far. Also notice that Han et al. (2015b) and Guo et al. (2016) find out it is necessary to fix the convolutional layers when pruning and retraining the fully-connected layer (and vice versa), otherwise the model cannot recover the pruning error. However, we don’t observe such difficulty in our experiments, as we simply prune and retrain all layers simultaneously. This also indicates that our pruning operation only brings small error and still preserves the model capacity.
  We further apply our method to a modern neural network architecture, VGG16. VGG16 is a deep network with 13 convolutional layers, 3 fully-connected layers and 138M parameters in total. Due to its large number of layers, the search space of per-layer compression ratio is exponentially larger than that of AlexNet. Therefore, conventional methods would have difficulties to tune and set the compression ratio for each of those 16 layers. On the contrary, our method can be used in a pushbutton-to-start manner, and automatically finds out appropriate ratios. We use the same pruning set-up as in AlexNet experiment, and achieve the state-of-the-art result as shown in Table 1.
sections.heading: 5.2 DOES MLPRUNE FIND REASONABLE PER-LAYER COMPRESSION RATIOS?
sections.text: We compare the compression ratios, layer by layer, between our method and the baselines, as shown in Table 2. The compression ratios of Han et al. (2015b) and Guo et al. (2016) are manually tuned to achieve good results. In contrast, our method automatically determines the compression ratio for each layer during the pruning process. It involves no time-consuming procedure for tuning hyper-parameters yet achieves better results. We also notice that our compression ratios are not the same as previous manually-tuned ratios, but share some similarities; layers with smaller compression ratios in our method also have smaller ratios in the baselines. Also, fully-connected layers are generally be pruned more severe than convolutional layers, which is in accord with the observation that fully-connected layers usually have more redundancies. These suggest that our method can find reasonable per-layer compression ratios according to the sensitivities of each layer. Detailed compression ratios for other architectures could be found in Appendix B.
sections.heading: 5.3 DOES MLPRUNE PRUNE SIMILAR PARAMETERS AS MAGNITUDE-BASED PRUNING?
sections.text: In section 5.2, we’ve seen MLPrune can automatically adjust how much to prune from different layers. In this section, we’ll investigate what parameters does MLPrune prune within one layer. Since both MLPrune and magnitude-based pruning can achieve good compression results, it’s intriguing
  to explore if MLPrune prune similar parameters as magnitude-based pruning, or they prune totally different parameters.
  Figure 1 shows the distribution of parameters’ magnitudes, coming from the first fully-connected layer of AlexNet. Before pruning, weight distribution is peaked at 0, and drops quickly as the absolute value increasing. This is very much like a gaussian distribution as what the weights were initialized from. Figure 1b shows the distribution after pruning using our method. It is obvious to see that parameters with magnitudes close to 0 (center region) are pruned away, indicating that the parameters regarded as having small impacts upon the loss by our method also usually have small magnitudes.
  We further explore the correlation between a parameter’s magnitude and its importance measured by our method. From figure 2, we can observe that: 1) A parameter’s absolute value indeed has a strong correlation with its importance computed by MLPrune. This explains why magnitude-based methods also achieve fairly good performance. Despite this strong correlation, directly compare the magnitudes from all layer will give different pruning result than our method: setting the pruning threshold as the median of parameters’ magnitudes in Figure 2a will prune 14% of conv5, while the median of our importance will prune 7%. This explains why magnitude-based pruning cannot do multi-layer pruning. 2) FC layer initially has smaller importance(larger redundancies) than convolutional layer measured by MLPrune, and thus are pruned more severely. As more and more parameters are pruned away(from Figure 2a to Figure 2b), the importance of FC layer becomes closer to that of convolutional layer, and both layers will be pruned with similar ratios. This shows that our method can dynamically adjust the compression ratio of each layer, as the pruning going on.
sections.heading: 6 CONCLUSION
sections.text: In this paper we have proposed an automated Multi-Layer Pruning (MLPrune) method to compress deep neural networks. Our approach exploits an efficient approximation of the Hessian based on the K-FAC method as our pruning criterion. It has much fewer hyper-parameters and can be used in an automated manner without costly test and trial processes. We have demonstrated the effectiveness of our method on several datasets and architectures and achieved state-of-the-art results. In the future, we plan to extend our method to prune coarser components such as filters, rather than individual weights, so as to speed up the network and save memory furthermore.
sections.heading: A PRUNING SETUP
sections.text: In this section, we provide the implementation details for experiments of AlexNet and VGG16.
  The AlexNet is first pre-trained using similar settings as in Caffe model zoo. A training image is first rescale so that the shorter edge is 256 with aspect ratio unchanged. Then a 227x227 image is randomly croped, followed by randomly flip horizontally or not. We do not apply color augmentation as in original paper (Krizhevsky et al., 2012). During testing, a center cropped image is fed into the network. SGD with momentum is applied for training. The batch-size is 256 and momentum is 0.9 for pre-training and all retraining stages.
  Table 3 shows the hyper-parameters used for retraining stages (We do not apply retraining when prune from 100% to 50% since the model performance doesn’t decrease before and after pruning). The learning rate schedule is exactly the same as pre-training, and doesn’t require any tuning. The weight decay is slightly decreased to regularize the model less, since a pruned model has smaller representation capacity. The only hyper-parameters need to tune during the pruning is the dropout rate. As more and more parameters are pruned away, the model will become under-fit rather than over-fit. Therefore, reduce the dropout rate accordingly will be helpful to get better model performance. (Han et al., 2015b) propose a heuristic formula to determine the appropriate dropout rate for magnitudebased pruning methods. However, applying their dropout rate on our model lead to severe over-fitting. For example, when compressed to 12.5%, their formula suggest dropout rate of 0.18%, while our model adopts 0.4%. One possible explanation is that magnitude-based pruning methods sometimes mistakenly prune away important connections. This will significantly harm the model capacity and need much smaller dropout rate.
  We adopt similar strategy for VGG16, namely using the same learning rate schedule, decrease and fix the weigh decay, and fine tune the dropout rate during pruning. The batch-size is 128 and momentum is 0.9 for all experiments. Again, we achieve state-of-the-art pruning result. Also notice that the dropout rate for VGG16 is larger than that of AlexNet, which might due to the fact that VGG16 has larger fully-connected layer, and thus pruning a fraction of it won’t affect the model capacity that much.
sections.heading: B PER-LAYER COMPRESSION RATIOS
sections.text: We compare the compression ratio, layer by layer, between our method and the baselines, as shown in Table 4. The compression ratios of Han et al. (2015b) and Guo et al. (2016) are manually tuned to achieve good results. In contrast, our method automatically determines the compression ratio for each layer during the pruning process. It involves no time-consuming procedure for tuning hyper-parameters yet achieves better results.