THE INCREDIBLE SHRINKING NEURAL NETWORK: NEW PERSPECTIVES
How much can pruning algorithms teach us about the fundamentals of learning representations in neural networks? A lot, it turns out. Neural network model compression has become a topic of great interest in recent years, and many different techniques have been proposed to address this problem. In general, this is motivated by the idea that smaller models typically lead to better generalization. At the same time, the decision of what to prune and when to prune necessarily forces us to confront our assumptions about how neural networks actually learn to represent patterns in data. In this work we set out to test several long-held hypotheses about neural network learning representations and numerical approaches to pruning. To accomplish this we first reviewed the historical literature and derived a novel algorithm to prune whole neurons (as opposed to the traditional method of pruning weights) from optimally trained networks using a second-order Taylor method. We then set about testing the performance of our algorithm and analyzing the quality of the decisions it made. As a baseline for comparison we used a first-order Taylor method based on the Skeletonization algorithm and an exhaustive brute-force serial pruning algorithm. Our proposed algorithm worked well compared to a first-order method, but not nearly as well as the brute-force method. Our error analysis led us to question the validity of many widely-held assumptions behind pruning algorithms in general and the trade-offs we often make in the interest of reducing computational complexity. We discovered that there is a straightforward way, however expensive, to serially prune 40-70% of the neurons in a trained network with minimal effect on the learning representation and without any re-training.
1 INTRODUCTION
In this work we propose and evaluate a novel algorithm for pruning whole neurons from a trained neural network without any re-training and examine its performance compared to two simpler methods. We then analyze the kinds of errors made by our algorithm and use this as a stepping off point to launch an investigation into the fundamental nature of learning representations in neural networks. Our results corroborate an insightful though largely forgotten observation by Mozer & Smolensky (1989a) concerning the nature of neural network learning. This observation is best summarized in a quotation from Segee & Carter (1991) on the notion of fault-tolerance in multilayer perceptron networks:
Contrary to the belief widely held, multilayer networks are not inherently fault tolerant. In fact, the loss of a single weight is frequently sufficient to completely
disrupt a learned function approximation. Furthermore, having a large number of weights does not seem to improve fault tolerance. [Emphasis added]
Essentially, Mozer & Smolensky (1989b) observed that during training neural networks do not distribute the learning representation evenly or equitably across hidden units. What actually happens is that a few, elite neurons learn an approximation of the input-output function, and the remaining units must learn a complex interdependence function which cancels out their respective influence on the network output. Furthermore, assuming enough units exist to learn the function in question, increasing the number of parameters does not increase the richness or robustness of the learned approximation, but rather simply increases the likelihood of overfitting and the number of noisy parameters to be canceled during training. This is evinced by the fact that in many cases, multiple neurons can be removed from a network with no re-training and with negligible impact on the quality of the output approximation. In other words, there are few bipartisan units in a trained network. A unit is typically either part of the (possibly overfit) input-output function approximation, or it is part of an elaborate noise cancellation task force. Assuming this is the case, most of the compute-time spent training a neural network is likely occupied by this arguably wasteful procedure of silencing superfluous parameters, and pruning can be viewed as a necessary procedure to “trim the fat.”
We observed copious evidence of this phenomenon in our experiments, and this is the motivation behind our decision to evaluate the pruning algorithms in this study on the simple criteria of their ability to trim neurons without any re-training. If we were to employ re-training as part of our evaluation criteria, we would arguably not be evaluating the quality of our algorithm’s pruning decisions per se but rather the ability of back-propagation trained networks to recover from faults caused by non-ideal pruning decisions, as suggested by the conclusions of Segee & Carter (1991) and Mozer & Smolensky (1989a). Moreover, as Fahlman & Lebiere (1989) discuss, due to the “herd effect” and “moving target” phenomena in back-propagation learning, the remaining units in a network will simply shift course to account for whatever error signal is re-introduced as a result of a bad pruning decision or network fault. So long as there are enough critical parameters to learn the function in question, a network can typically recover faults with additional training. This limits the conclusions we can draw about the quality of our pruning criteria when we employ re-training.
In terms of removing units without re-training, what we discovered is that predicting the behavior of a network when a unit is to be pruned is very difficult, and most of the approximation techniques put forth in existing pruning algorithms do not fare well at all when compared to a brute-force search. To begin our discussion of how we arrived at our algorithm and set up our experiments, we review of the existing literature.
2 LITERATURE REVIEW
Pruning algorithms, as comprehensively surveyed by Reed (1993), are a useful set of heuristics designed to identify and remove elements from a neural network which are either redundant or do not significantly contribute to the output of the network. This is motivated by the observed tendency of neural networks to overfit to the idiosyncrasies of their training data given too many trainable parameters or too few input patterns from which to generalize, as stated by Chauvin (1990).
Network architecture design and hyperparameter selection are inherently difficult tasks typically approached using a few well-known rules of thumb, e.g. various weight initialization procedures, choosing the width and number of layers, different activation functions, learning rates, momentum, etc. Some of this “black art” appears unavoidable. For problems which cannot be solved using linear threshold units alone, Baum & Haussler (1989) demonstrate that there is no way to precisely determine the appropriate size of a neural network a priori given any random set of training instances. Using too few neurons seems to inhibit learning, and so in practice it is common to attempt to overparameterize networks initially using a large number of hidden units and weights, and then prune or compress them afterwards if necessary. Of course, as the old saying goes, there’s more than one way to skin a neural network.
2.1 NON-PRUNING BASED GENERALIZATION & COMPRESSION TECHNIQUES
The generalization behavior of neural networks has been well studied, and apart from pruning algorithms many heuristics have been used to avoid overfitting, such as dropout (Srivastava et al.
(2014)), maxout (Goodfellow et al. (2013)), and cascade correlation (Fahlman & Lebiere (1989)), among others. Of course, while cascade correlation specifically tries to construct of minimal networks, many techniques to improve network generalization do not explicitly attempt to reduce the total number of parameters or the memory footprint of a trained network per se.
Model compression often has benefits with respect to generalization performance and the portability of neural networks to operate in memory-constrained or embedded environments. Without explicitly removing parameters from the network, weight quantization allows for a reduction in the number of bytes used to represent each weight parameter, as investigated by Balzer et al. (1991), Dundar & Rose (1994), and Hoehfeld & Fahlman (1992).
A recently proposed method for compressing recurrent neural networks (Prabhavalkar et al. (2016)) uses the singular values of a trained weight matrix as basis vectors from which to derive a compressed hidden layer. Øland & Raj (2015) successfully implemented network compression through weight quantization with an encoding step while others such as Han et al. (2016) have tried to expand on this by adding weight-pruning as a preceding step to quantization and encoding.
In summary, we can say that there are many different ways to improve network generalization by altering the training procedure, the objective error function, or by using compressed representations of the network parameters. But these are not, strictly speaking, examples of techniques to reduce the number of parameters in a network. For this we must employ some form of pruning criteria.
2.2 PRUNING TECHNIQUES
If we wanted to continually shrink a neural network down to minimum size, the most straightforward brute-force way to do it is to individually switch each element off and measure the increase in total error on the training set. We then pick the element which has the least impact on the total error, and remove it. Rinse and repeat. This is extremely computationally expensive, given a reasonably large neural network and training set. Alternatively, we might accomplish this using any number of much faster off-the-shelf pruning algorithms, such as Skeletonization (Mozer & Smolensky (1989a)), Optimal Brain Damage (LeCun et al. (1989)), or later variants such as Optimal Brain Surgeon (Hassibi & Stork (1993)). In fact, we borrow much of our inspiration from these algorithms, with one major variation: Instead of pruning individual weights, we prune entire neurons, thereby eliminating all of their incoming and outgoing weight parameters in one go, resulting in more memory saved, faster.
The algorithm developed for this paper is targeted at reducing the total number of neurons in a trained network, which is one way of reducing its computational memory footprint. This is often a desirable criteria to minimize in the case of resource-constrained or embedded devices, and also allows us to probe the limitations of pruning down to the very last essential network elements. In terms of generalization as well, we can measure the error of the network on the test set as each element is sequentially removed from the network. With an oracle pruning algorithm, what we expect to observe is that the output of the network remains stable as the first few superfluous neurons are removed, and as we start to bite into the more crucial members of the function approximation, the error should start to rise dramatically. In this paper, the brute-force approach described at the beginning of this section serves as a proxy for an oracle pruning algorithm.
One reason to choose to rank and prune individual neurons as opposed to weights is that there are far fewer elements to consider. Furthermore, the removal of a single weight from a large network is a drop in the bucket in terms of reducing a network’s core memory footprint. If we want to reduce the size of a network as efficiently as possible, we argue that pruning neurons instead of weights is more efficient computationally as well as practically in terms of quickly reaching a hypothetical target reduction in memory consumption. This approach also offers downstream applications a realistic expectation of the minimal increase in error resulting from the removal of a specified percentage of neurons. Such trade-offs are unavoidable, but performance impacts can be limited if a principled approach is used to find the best candidate neurons for removal.
It is well known that too many free parameters in a neural network can lead to overfitting. Regardless of the number of weights used in a given network, as Segee & Carter (1991) assert, the representation of a learned function approximation is almost never evenly distributed over the hidden units, and thus the removal of any single hidden unit at random can actually result in a network fault. Mozer & Smolensky (1989b) argue that only a subset of the hidden units in a neural network actually
latch on to the invariant or generalizing properties of the training inputs, and the rest learn to either mutually cancel each other’s influence or begin overfitting to the noise in the data. We leverage this idea in the current work to rank all neurons in pre-trained networks based on their effective contributions to the overall performance. We then remove the unnecessary neurons to reduce the network’s footprint. Through our experiments we not only concretely validate the theory put forth by Mozer & Smolensky (1989b) but we also successfully build on it to prune networks to 40 to 60 % of their original size without any major loss in performance.
3 PRUNING NEURONS TO SHRINK NEURAL NETWORKS
As discussed in Section 1 our aim is to leverage the highly non-uniform distribution of the learning representation in pre-trained neural networks to eliminate redundant neurons, without focusing on individual weight parameters. Taking this approach enables us to remove all the weights (incoming and outgoing) associated with a non-contributing neuron at once. We would like to note here that in an ideal scenario, based on the neuron interdependency theory put forward by Mozer & Smolensky (1989a), one would evaluate all possible combinations of neurons to remove (one at a time, two at a time, three at a time and so forth) to find the optimal subset of neurons to keep. This is computationally unacceptable, and so we will only focus on removing one neuron at a time and explore more “greedy” algorithms to do this in a more efficient manner.
The general approach taken to prune an optimally trained neural network here is to create a ranked list of all the neurons in the network based off of one of the 3 proposed ranking criteria: a brute force approximation, a linear approximation and a quadratic approximation of the neuron’s impact on the output of the network. We then test the effects of removing neurons on the accuracy and error of the network. All the algorithms and methods presented here are easily parallelizable as well.
One last thing to note here before moving forward is that the methods discussed in this section involve some non-trivial derivations which are beyond the scope of this paper. We are more focused on analyzing the implications of these methods on our understanding of neural network learning representations. However, a complete step-by-step derivation and proof of all the results presented is provided in the Supplementary Material as an Appendix.
3.1 BRUTE FORCE REMOVAL APPROACH
This is perhaps the most naive yet the most accurate method for pruning the network. It is also the slowest and hence possibly unusable on large-scale neural networks with thousands of neurons. This method explicitly evaluates each neuron in the network. The idea is to manually check the effect of every single neuron on the output. This is done by running a forward propagation on the validation set K times (where K is the total number of neurons in the network), turning off exactly one neuron each time (keeping all other neurons active) and noting down the change in error. Turning a neuron off can be achieved by simply setting its output to 0. This results in all the outgoing weights from that neuron being turned off. This change in error is then used to generate the ranked list.
3.2 TAYLOR SERIES REPRESENTATION OF ERROR
Let us denote the total error from the optimally trained neural network for any given validation dataset by E. E can be seen as a function of O, where O is the output of any general neuron in the network. This error can be approximated at a particular neuron’s output (say Ok) by using the 2nd order Taylor Series as,
Ê(O) ≈ E(Ok) + (O −Ok) · ∂E
∂O ∣∣∣∣ Ok + 0.5 · (O −Ok)2 · ∂2E ∂O2 ∣∣∣∣ Ok , (1)
When a neuron is pruned, its output O becomes 0.
Replacing O by Ok in equation 1 shows us that the error is approximated perfectly by equation 1 at Ok. So:
∆Ek = Ê(0)− Ê(Ok) = −Ok · ∂E
∂O ∣∣∣∣ Ok + 0.5 ·O2k · ∂2E ∂O2 ∣∣∣∣ Ok , (2)
where ∆Ek is the change in the total error of the network when exactly one neuron (k) is turned off. Most of the terms in this equation are fairly easy to compute, as we have Ok already from the activations of the hidden units and we already compute ∂E∂O |Ok for each training instance during backpropagation. The ∂
2E ∂O2 |Ok terms are a little more difficult to compute. This is derived in the
appendix and summarized in the sections below.
3.2.1 LINEAR APPROXIMATION APPROACH
We can use equation 2 to get the linear error approximation of the change in error due to the kth neuron being turned off and represent it as ∆E1k as follows:
∆E1k = −Ok · ∂E
∂O ∣∣∣∣ Ok
(3)
The derivative term above is the first-order gradient which represents the change in error with respect to the output a given neuron. This term can be collected during back-propagation. As we shall see further in this section, linear approximations are not reliable indicators of change in error but they provide us with an interesting basis for comparison with the other methods discussed in this paper.
3.2.2 QUADRATIC APPROXIMATION APPROACH
As above, we can use equation 2 to get the quadratic error approximation of the change in error due to the kth neuron being turned off and represent it as ∆E2k as follows:
∆E2k = −Ok · ∂E
∂O ∣∣∣∣ Ok + 0.5 ·O2k · ∂2E ∂O2 ∣∣∣∣ Ok
(4)
The additional second-order gradient term appearing above represents the quadratic change in error with respect to the output of a given neuron. This term can be generated by performing backpropagation using second order derivatives. Collecting these quadratic gradients involves some non-trivial mathematics, the entire step-by-step derivation procedure of which is provided in the Supplementary Material as an Appendix.
3.3 PROPOSED PRUNING ALGORITHM
Figure 1 shows a random error function plotted against the output of any given neuron. Note that this figure is for illustration purposes only. The error function is minimized at a particular value of the neuron output as can be seen in the figure. The process of training a neural network is essentially the process of finding these minimizing output values for all the neurons in the network. Pruning this particular neuron (which translates to getting a zero output from it will result in a change in the total overall error. This change in error is represented by distance between the original minimum error (shown by the dashed line) and the top red arrow. This neuron is clearly a bad candidate for removal since removing it will result in a huge error increase.
The straight red line in the figure represents the first-order approximation of the error using Taylor Series as described before while the parabola represents a second-order approximation. It can be clearly seen that the second-order approximation is a much better estimate of the change in error.
One thing to note here is that it is possible in some cases that there is some thresholding required when trying to approximate the error using the 2nd order Taylor Series expansion. These cases might arise when the parabolic approximation undergoes a steep slope change. To take into account such cases, mean and median thresholding were employed, where any change above a certain threshold was assigned a mean or median value respectively.
Two pruning algorithms are proposed here. They are different in the way the neurons are ranked but both of them use ∆Ek, the approximation of the change in error as the basis for the ranking. ∆Ek can be calculated using the Brute Force method, or one of the two Taylor Series approximations discussed previously.
The first step in both the algorithms is to decide a stopping criterion. This can vary depending on the application but some intuitive stopping criteria can be: maximum number of neurons to remove, percentage scaling needed, maximum allowable accuracy drop etc.
3.3.1 ALGORITHM I: SINGLE OVERALL RANKING
The complete algorithm is shown in Algorithm 1. The idea here is to generate a single ranked list based on the values of ∆Ek. This involves a single pass of second-order back-propagation (without weight updates) to collect the gradients for each neuron. The neurons from this rank-list (with the lowest values of ∆Ek) are then pruned according to the stopping criterion decided. We note here that this algorithm is intentionally naive and is used for comparison only.
Data: optimally trained network, training set Result: A pruned network initialize and define stopping criterion ; perform forward propagation over the training set ; perform second-order back-propagation without updating weights and collect linear and quadratic
gradients ; rank the remaining neurons based on ∆Ek; while stopping criterion is not met do
remove the last ranked neuron ; end
Algorithm 1: Single Overall Ranking
3.3.2 ALGORITHM II: ITERATIVE RE-RANKING
In this greedy variation of the algorithm (Algorithm 2), after each neuron removal, the remaining network undergoes a single forward and backward pass of second-order back-propagation (without weight updates) and the rank list is formed again. Hence, each removal involves a new pass through
the network. This method is computationally more expensive but takes into account the dependencies the neurons might have on one another which would lead to a change in error contribution every time a dependent neuron is removed.
Data: optimally trained network, training set Result: A pruned network initialize and define stopping criterion ; while stopping criterion is not met do
perform forward propagation over the training set ; perform second-order back-propagation without updating weights and collect linear and
quadratic gradients ; rank the remaining neurons based on ∆Ek ; remove the worst neuron based on the ranking ;
end Algorithm 2: Iterative Re-Ranking
4 EXPERIMENTAL RESULTS
4.1 EXAMPLE REGRESSION PROBLEM
This problem serves as a quick example to demonstrate many of the phenomena described in previous sections. We trained two networks to learn the cosine function, with one input and one output. This is a task which requires no more than 11 sigmoid neurons to solve entirely, and in this case we don’t care about overfitting because the cosine function has a precise definition. Furthermore, the cosine function is a good toy example because it is a smooth continuous function and, as demonstrated by Nielsen (2015), if we were to tinker directly with the weights and bias parameters of the network, we could allocate individual units within the network to be responsible for constrained ranges of inputs, similar to a basis spline function with many control points. This would distribute the learned function approximation evenly across all hidden units, and thus we have presented the network with a problem in which it could productively use as many hidden units as we give it. In this case, a pruning algorithm would observe a fairly consistent increase in error after the removal of each successive unit. In practice however, regardless of the number of experimental trials, this is not what happens. The network will always use 10-11 hidden units and leave the rest to cancel each other’s influence.
Figure 2 shows two graphs. Both graphs demonstrate the use of the iterative re-ranking algorithm and the comparative performance of the brute-force pruning method (in blue), the first order method (in green), and the second order method (in red). The graph on the left shows the performance of these algorithms starting from a network with two layers of 10 neurons (20 total), and the graph on the right shows a network with two layers of 50 neurons (100 total).
In the left graph, we see that the brute-force method shows a graceful degradation, and the error only begins to rise sharply after 50% of the total neurons have been removed. The error is basically constant up to that point. In the first and second order methods, we see evidence of poor decision making in the sense that both made mistakes early on, which disrupted the output function approximation. The first order method made a large error early on, though we see after a few more neurons were removed this error was corrected somewhat (though it only got worse from there). This is direct evidence of the lack of fault tolerance in a trained neural network. This phenomenon is even more starkly demonstrated in the second order method. After making a few poor neuron removal decisions in a row, the error signal rose sharply, and then went back to zero after the 6th neuron was removed. This is due to the fact that the neurons it chose to remove were trained to cancel each others’ influence within a localized part of the network. After the entire group was eliminated, the approximation returned to normal. This can only happen if the output function approximation is not evenly distributed over the hidden units in a trained network.
This phenomenon is even more starkly demonstrated in the graph on the right. Here we see the first order method got “lucky” in the beginning and made decent decisions up to about the 40th removed neuron. The second order method had a small error in the beginning which it recovered from gracefully and proceeded to pass the 50 neuron point before finally beginning to unravel. The brute force method, in sharp contrast, shows little to no increase in error at all until 90% of the neurons in the network have been obliterated. Clearly first and second order methods have some value in that they do not make completely arbitrary choices, but the brute force method is far better at this task.
This also demonstrates the sharp dualism in neuron roles within a trained network. These networks were trained to near-perfect precision and each pruning method was applied without any re-training of any kind. Clearly, in the case of the brute force or oracle method, up to 90% of the network can be completely extirpated before the output approximation even begins to show any signs of degradation. This would be impossible if the learning representation were evenly or equitably distributed. Note, for example, that the degradation point in both cases is approximately the same. This example is not a real-world application of course, but it brings into very clear focus the kind of phenomena we will discuss in the following sections.
4.2 RESULTS ON MNIST DATASET
For all the results presented in this section, the MNIST database of Handwritten Digits by LeCun & Cortes (2010) was used. It is worth noting that due to the time taken by the brute force algorithm we rather used a 5000 image subset of the MNIST database in which we have normalized the pixel values between 0 and 1.0, and compressed the image sizes to 20x20 images rather than 28x28, so the starting test accuracy reported here appears higher than those reported by LeCun et al. We do not believe that this affects the interpretation of the presented results because the basic learning problem does not change with a larger dataset or input dimension.
4.3 PRUNING A 1-LAYER NETWORK
The network architecture in this case consisted of 1 layer, 100 neurons, 10 outputs, logistic sigmoid activations, and a starting test accuracy of 0.998.
4.3.1 SINGLE OVERALL RANKING ALGORITHM
We first present the results for a single-layer neural network in Figure 3, using the Single Overall algorithm (Algorithm 1) as proposed in Section 3. (We again note that this algorithm is intentionally naive and is used for comparison only. Its performance should be expected to be poor.) After training, each neuron is assigned its permanent ranking based on the three criteria discussed previously: A brute force “ground truth” ranking, and two approximations of this ranking using first and second order Taylor estimations of the change in network output error resulting from the removal of each neuron.
An interesting observation here is that with only a single layer, no criteria for ranking the neurons in the network (brute force or the two Taylor Series variants) using Algorithm 1 emerges superior, indicating that the 1st and 2nd order Taylor Series methods are actually reasonable approximations
of the brute force method under certain conditions. Of course, this method is still quite bad in terms of the rate of degradation of the classification accuracy and in practice we would likely follow Algorithm 2 which is takes into account Mozer & Smolensky (1989a)’s observations stated in the Related Work section. The purpose of the present investigation, however, is to demonstrate how much of a trained network can be theoretically removed without altering the network’s learned parameters in any way.
4.3.2 ITERATIVE RE-RANKING ALGORITHM
In Figure 4 we present our results using Algorithm 2 (The iterative re-ranking Algorithm) in which all remaining neurons are re-ranked after each successive neuron is switched off. We compute the same brute force rankings and Taylor series approximations of error deltas over the remaining active neurons in the network after each pruning decision. This is intended to account for the effects of cancelling interactions between neurons.
There are 2 key observations here. Using the brute force ranking criteria, almost 60% of the neurons in the network can be pruned away without any major loss in performance. The other noteworthy observation here is that the 2nd order Taylor Series approximation of the error performs consistently better than its 1st order version, in most situations, though Figure 21 is a poignant counter-example.
4.3.3 VISUALIZATION OF ERROR SURFACE & PRUNING DECISIONS
As explained in Section 3, these graphs are a visualization of the error surface of the network output with respect to the neurons chosen for removal using each of the 3 ranking criteria, represented in
intervals of 10 neurons. In each graph, the error surface of the network output is displayed in log space (left) and in real space (right) with respect to each candidate neuron chosen for removal. We create these plots during the pruning exercise by picking a neuron to switch off, and then multiplying its output by a scalar gain value α which is adjusted from 0.0 to 10.0 with a step size of 0.001. When the value of α is 1.0, this represents the unperturbed neuron output learned during training. Between 0.0 and 1.0, we are graphing the literal effect of turning the neuron off (α = 0), and when α > 1.0 we are simulating a boosting of the neuron’s influence in the network, i.e. inflating the value of its outgoing weight parameters.
We graph the effect of boosting the neuron’s output to demonstrate that for certain neurons in the network, even doubling, tripling, or quadrupling the scalar output of the neuron has no effect on the overall error of the network, indicating the remarkable degree to which the network has learned to ignore the value of certain parameters. In other cases, we can get a sense of the sensitivity of the network’s output to the value of a given neuron when the curve rises steeply after the red 1.0 line. This indicates that the learned value of the parameters emanating from a given neuron are relatively important, and this is why we should ideally see sharper upticks in the curves for the later-removed neurons in the network, that is, when the neurons crucial to the learning representation start to be picked off. Some very interesting observations can be made in each of these graphs.
Remember that lower is better in terms of the height of the curve and minimal (or negative) horizontal change between the vertical red line at 1.0 (neuron on, α = 1.0) and 0.0 (neuron off, α = 0.0) is indicative of a good candidate neuron to prune, i.e. there will be minimal effect on the network output when the neuron is removed.
4.3.4 VISUALIZATION OF BRUTE FORCE PRUNING DECISIONS
In Figure ??, we notice how low to the floor and flat most of the curves are. It’s only until the 90th removed neuron that we see a higher curve with a more convex shape (clearly a more sensitive, influential piece of the network).
4.3.5 VISUALIZATION OF 1ST ORDER APPROXIMATION PRUNING DECISIONS
It can be seen in Figure 6 that most choices seem to have flat or negatively sloped curves, indicating that the first order approximation seems to be pretty good, but examining the brute force choices shows they could be better.
4.3.6 VISUALIZATION OF 2ND ORDER APPROXIMATION PRUNING DECISIONS
The method in Figure 7 looks similar to the brute force method choices, though clearly not as good (they’re more spread out). Notice the difference in convexity between the 2nd and 1st order method
choices. It’s clear that the first order method is fitting a line and the 2nd order method is fitting a parabola in their approximation.
4.4 PRUNING A 2-LAYER NETWORK
The network architecture in this case consisted of 2 layers, 50 neurons per layer, 10 outputs, logistic sigmoid activations, and a starting test accuracy of 1.000.
4.4.1 SINGLE OVERALL RANKING ALGORITHM
Figure 8 shows the pruning results for Algorithm 1 on a 2-layer network. The ranking procedure is identical to the one used to generate Figure 3. (We again note that this algorithm is intentionally naive and is used for comparison only. Its performance should be expected to be poor.)
Unsurprisingly, a 2-layer network is harder to prune because a single overall ranking will never capture the interdependencies between neurons in different layers. It makes sense that this is worse
than the performance on the 1-layer network, even if this method is already known to be bad, and we’d likely never use it in practice.
4.4.2 ITERATIVE RE-RANKING ALGORITHM
Figure 9 shows the results from using Algorithm 2 on a 2-layer network. We compute the same brute force rankings and Taylor series approximations of error deltas over the remaining active neurons in the network after each pruning decision used to generate Figure 4. Again, this is intended to account for the effects of cancelling interactions between neurons.
It is clear that it becomes harder to remove neurons 1-by-1 with a deeper network (which makes sense because the neurons have more interdependencies in a deeper network), but we see an overall better performance with 2nd order method vs. 1st order, except for the first 20% of the neurons (but this doesn’t seem to make much difference for classification accuracy.)
Perhaps a more important observation here is that even with a more complex network, it is possible to remove up to 40% of the neurons with no major loss in performance which is clearly illustrated by the brute force curve. This shows the clear potential of an ideal pruning technique and also shows how inconsistent 1st and 2nd order Taylor Series approximations of the error can be as ranking criteria.
4.4.3 VISUALIZATION OF ERROR SURFACE & PRUNING DECISIONS
As seen in the case of a single layered network, these graphs are a visualization the error surface of the network output with respect to the neurons chosen for removal using each algorithm, represented in intervals of 10 neurons.
4.4.4 VISUALIZATION OF BRUTE FORCE PRUNING DECISIONS
In Figure 10, it is clear why these neurons got chosen, their graphs clearly show little change when neuron is removed, are mostly near the floor, and show convex behaviour of error surface, which argues for the rationalization of using 2nd order methods to estimate difference in error when they are turned off.
4.4.5 VISUALIZATION OF 1ST ORDER APPROXIMATION PRUNING DECISIONS
Drawing a flat line at the point of each neurons intersection with the red vertical line (no change in gain) shows that the 1st derivative method is actually accurate for estimation of change in error in these cases, but still ultimately leads to poor decisions.
4.4.6 VISUALIZATION OF 2ND ORDER APPROXIMATION PRUNING DECISIONS
Clearly these neurons are not overtly poor candidates for removal (error doesn’t change much between 1.0 & zero-crossing left-hand-side), but could be better (as described above in the brute force Criterion discussion).
4.5 INVESTIGATION OF PRUNING PERFORMANCE WITH IMPERFECT STARTING CONDITIONS
In our experiments thus far we have tacitly assumed that we start with a network which has learned an “optimal” representation of the training objective, i.e. it has been trained to the point where we accept its performance on the test set. Here we explore what happens when we prune with a sub-optimal starting network.
If the assumptions of this paper regarding the nature of neural network learning are correct, we expect that two processes are essentially at work during back-propagation training. First, we expect that the neurons which directly participate in the fundamental learning representation (even if redundantly) work together to reduce error on the training data. Second, we expect that neurons which do not directly participate in the learning representation work to cancel each other’s negative influence. Furthermore, we expect that these two groups are essentially distinct, as evinced by the fact that multiple neurons can often be removed as a group with little to no effect on the network output. Some non-trivial portion of the training time, then, is spent doing work which has nothing intrinsically to do with the learning representation and essentially functions as noise cancellation.
If this is the case, when we attempt to prune a network which has not fully canceled the noisy influence of extraneous or redundant units, we might expect to see the error actually improve after removing a few bad apples. This is in fact what we observe, as demonstrated in the following experiments.
For each experiment in this section we trained with the full MNIST training set (LeCun & Cortes (2010)), uncompressed and without any data normalization. We trained three different networks to learn to distinguish a single handwritten digit from the rest of the data. The network architectures were each composed of 784 inputs, 1 hidden layer with 100 neurons, and 2 soft-max outputs; one to say yes, and the other to say no. These networks were trained to distinguish the digits 0, 1, and 2, and their respective starting accuracies were a sub-optimal 0.9757, 0.9881, and 0.9513. Finally, we only consider the iterative re-ranking algorithm, as the single overall ranking algorithm is clearly nonviable.
4.5.1 MNIST SINGLE DIGIT CLASSIFICATION: DIGIT 0
Figure 13 shows the degradation in squared error after removing neurons from a network trained to distinguish the digit 0. What we observe is that the first and second order methods both fail in different ways, though clearly the second order method makes better decisions overall. The first order method explodes spectacularly in the first few iterations. The brute force method, in stark contrast, actually improves in the first few iterations, and remains essentially flat until around the 60% mark, at which point it begins to gradually increase and meet the other curves.
The behavior of the brute force method here demonstrates that the network was essentially working to cancel the effect of a few bad neurons when the training convergence criteria were met, i.e. the network was no longer able to make progress on the training set. After removing these neurons during pruning, the output improved. We can investigate this by looking at the error surface with respect to the neurons chosen for removal by each method in turn. Below in Figure 14 is the graph of the brute force method.
Figure 14 shows an interesting phenomenon, which we will see in later experiments as well. The high blue curve corresponding to neuron 0 is negatively sloped in the beginning and clearly after removing this neuron, the output will improve. The rest of the curves, in correspondence with the squared error degradation curve above, are mostly flat and tightly layered together, indicating that they are good neurons to remove.
In Figure 15 below, we observe a stark contrast to this. The curves corresponding to neurons 0 and 10 are mostly flat, and fairly lower than the rest, though clearly a mistake was made early on and the rest of the curves are clearly bad choices. In all of these cases however, we see that the curves are
easily approximated with a straight line and so the first order method may have been fairly accurate in its predictions, even though it still made poor decisions.
Figure 15 is an example of how things can go south once a few bad mistakes are made at the outset. Figure 16 shows a much better set of choices made by the second order method, though clearly not as good as the brute force method. The log-space plots make it a bit easier to see the difference between the brute force and second order methods in Figures 14 and 16, respectively.
4.5.2 MNIST SINGLE DIGIT CLASSIFICATION: DIGIT 1
Examining Figure 17, we see a much starker example of the previous phenomenon, in which the brute force method continues to improve the performance of the network after removing 80% of the neurons in the network. The first and second order methods fail early and proceed in fits and starts (clearly demonstrating evidence of interrelated groups of noise-canceling neurons), and never fully recover. It should be noted that it would be impossible to see curves like this if neural networks evenly distributed the learning representation evenly or equitably over their hidden units.
One of the most striking things about the blue curve in Figure 17 is the fact that the network never drops below its starting error until it crosses the 80% mark, indicating that only 20% of the neurons in this network are actually essential to the learning the training objective. In this sense, we can only
wonder how much of the training time was spent winnowing the error out of the remaining 80% of the network.
In Figures 18, 19 and 20 we can examine the choices made by the respective methods. The brute force method serves as our example of a near-optimal pruning regimen, and the rest are first and second order approximations of this. Small differences, clearly, can lead to large effects on the network output as shown in Figure 17.
4.5.3 MNIST SINGLE DIGIT CLASSIFICATION: DIGIT 2
Figure 21 is an interesting case because it shatters our confidence in the reliability of the second order method to make good pruning decisions, and further demonstrates the phenomenon of how much the error can improve if the right neurons are removed after training gets stuck. In this case, though still a poor performance overall, the first order method vastly outperforms the second order method.
Figure 22 shows us a clear example of the first element to remove having a negative error slope, and improving the output as a result. The rest of the pruning decisions are reasonable. Comparing with the blue curve in Figure 21, we see the correspondence between the first pruning decision improving the output, and the remaining pruning decisions keeping the output fairly flat. Clearly, however, there isn’t much room to get worse given our starting point with a sub-optimal network, and we see that the ending sum of squared errors is not much higher than the starting point. At the same time, we can still see the contrast in performance if we make optimal pruning decisions, and most of the neurons in this network were clearly doing nothing.
In Figure 23, we see a mixed bag in which the decisions are clearly sub-optimal, though much better than Figure 24, in which we can observe how a bad first decision essentially ruined the network for good. The jagged edges of the red curve in Figure 21 correspond with the positive and negative
slopes of the cluster of bad pruning decisions in 24. Once again, these are not necessarily bad decisions, but the starting point is already bad and this cannot be recovered without re-training the network.
4.5.4 ASIDE: IMPLICATIONS OF THIS EXPERIMENT
From the three examples above, we see that in each case, starting from a sub-optimal network, a brute force removal technique consistently improves performance for the first few pruning iterations, and the sum of squared errors does not degrade beyond the starting point until around 60-80% of the neurons have been removed. This is only possible if we have an essentially strict dichotomy between the roles of different neurons during training. If the network needs only 20-40% of the neurons it began with, the training process is essentially dominated by the task of canceling the residual noise of redundant neurons. Furthermore, the network can get stuck in training with redundant units and distort the final output. This is strong evidence of our thesis that the learning representation is neither equitably nor evenly distributed and that most of the neurons which do not directly participate in the learning representation can be removed without any retraining.
4.6 EXPERIMENTS ON TOY DATASETS
As can be seen from the experiments on MNIST, even though the 2nd-order approximation criterion is consistently better than 1st-order, its performance is not nearly as good as brute force based rank-
ing, especially beyond the first layer. What is interesting to note is that from some other experiments conducted on toy datasets (predicting whether a given point would lie inside a given shape on the Cartesian plane), the performance of the 2nd-order method was found to be exceptionally good and produced results very close to the brute force method. The 1st-order method, as expected, performed poorly here as well. Some of these results are illustrated in Figure 25.
5 CONCLUSIONS & FUTURE WORK
In conclusion, we must first re-assert that we do not present this work as a bench-marking study of the algorithm we derived and tested. We have merely used this algorithm as a jumping off point to investigate the nature of learning representations in neural networks. What we discovered is that first and second order methods do not make particularly good pruning decisions, and can get hopelessly lost after making a bad pruning decision resulting in a network fault. Furthermore, the brute-force algorithm does surprisingly well, despite being computationally expensive. This method does so well in fact, we argue that further investigation is warranted to make this algorithm computationally tractable, though we do not speculate on how that should be done here.
We also observed strong evidence for the hypotheses of Mozer & Smolensky (1989a) regarding the “dualist” nature of hidden units, i.e. that learning representations are divided between units which either participate in the output approximation or learn to cancel each others influence. This suggests that neural networks may in fact learn a minimal network implicitly, though we cannot say for sure that this is the case without further investigation. A necessary experiment to this end would be to compare the size of network constructed using cascade correlation (Fahlman & Lebiere (1989)) and compare it to the results described herein.
We have presented a novel algorithm for pruning whole neurons from a trained neural network using a second-order Taylor series approximation of the change in error resulting from the removal a given neuron as a pruning criteria. We compared this method to a first order method and a bruteforce serial removal method which exhaustively found the next best single neuron to remove at each stage. Our algorithm relies on a combination of assumptions similar to the ones made by Mozer & Smolensky (1989a) and LeCun et al. (1989) in the formulation of the Skeletonization and Optimal Brain Damage algorithms.
First, we assumed that the error function with respect to each individual neuron can be approximated with a straight line or more precisely with a parabola. Second, for second derivative terms we consider only the diagonal elements of the Hessian matrix, i.e. we assume that each neuron-weight connection can be treated independently of the other elements in the network. Third, we assumed that pruning could be done in a serial fashion in which we find the single least productive element in the network, remove it, and move on. We found that all of these assumptions are deeply flawed in the sense that the true relevance of a neuron can only be partially approximated by a first or second order method, and only at certain stages of the pruning process.
For most problems, these methods can usually remove between 10-30% of the neurons in a trained network, but beyond this point their reliability breaks down. For certain problems, none of the described methods seem to perform very well, though for obvious reasons the brute-force method always exhibits the best results. The reason for this is that the error function with respect to each hidden unit is more complex than a simple second-order Taylor series can approximate. Furthermore, we have not directly taken into account the interdependence of elements within a network, though the work of Hassibi & Stork (1993) could provide some guidance in this regard. This is another critical issue to investigate in the future.
Re-training may help in this regard. We freely admit that our algorithm does not use re-training to recover from errors made in pruning decisions. We argue that evaluating a network pruning algorithm using re-training does not allow us to make fair comparisons between the kinds of decisions made by these algorithms. Neural networks are very good at recovering from the removal of individual elements with re-training and so this compensates for sub-optimal pruning criteria.
We have observed that pruning whole neurons from an optimally trained network without major loss in performance is not only possible but also enables compressing networks to 40-70% of their original size, which is of great importance in constrained memory environments like embedded devices. We cite the results of our experiments using the brute force criterion as evidence of this conclusion. However expensive, it would be extremely easy to parallelize this method, or potentially approximate it using a subset of the training data to decide which neurons to prune. This avoids the problem of trying to approximate the importance of a unit and potentially making a mistake.
It would also be interesting to see how these methods perform on deeper networks and on some other popular and real world datasets. In our case, on the MNIST dataset, we observed that it was more difficult to prune neurons from a deeper network than from one with a single layer. We should expect
this trend to continue as networks get deeper and deeper, which also calls into further question the reliability of the described first and second order methods. We did investigate the order in which neurons were plucked from each layer of the networks and we found that the brute force method primarily removes neurons from the deepest layer of the network first, but there was no obvious pattern in layer preference for the other two methods.
Our experiments using the visualization of error surfaces and pruning decisions concretely establish the fact that not all neurons in a network contribute to its performance in the same way, and the observed complexity of these functions demonstrates limitations of the approximations we used.
Finally, we encourage the readers of this work to take these results into consideration when making decisions as to which methods to use to improve network generalization or compress their models. It should be remembered that various heuristics may perform well in practice for reasons which are in fact orthogonal to the accepted justifications given by their proponents.

The paper introduces a new pruning method for neural networks based on the second-order Taylor expansion and compares the results against a first-order method and brute-force pruning. It performs experiments of the three methods on several toy examples - including a two-layer network on MNIST - and shows that the second-order method behaves much worse then the brute-force baseline. In addition, from the success of the brute-force pruning the authors conclude that the hypothesis of Mozer et al - that neurons either contribute to performance or cancel out the effect of other neurons - is probably correct.
The authors put in considerable effort to explain all details of the paper clearly and at length, so the content of the paper is accessible even to people novel to pruning methods. Additionally, the authors have very carefully answered all questions that were coming up through the pre-review and have been very responsive.
My major criticism is that the paper lacks focus, does not have a concrete conclusion and does not explain what it adds to the literature. To make this apparent, I here summarise each paragraph of the conclusion section:
Paragraph 1: We do not benchmark / Pruning methods do not fare well against brute-force baseline / Some evidence for hypothesis of Mozer & Smolensky, but further investigation needed
Paragraph 2: Introduced 2nd order Taylor method / Does not fare well against baseline
Paragraph 3: Re-training may help but is not fair
Paragraph 4: Brute-force can prune 40-70% in shallow networks
Paragraph 5: Brute-force less effective in deep networks
Paragraph 6: Not all neurons contribute equally to performance of network
The title of the paper and answers of the authors to the pre-review questions seemed to strongly suggest that the paper is not about the new second-order method, is not about benchmarking pruning algorithms but is instead about the learnt representations. But only two or three sentences in the conclusion, and no sentence in the part on results in the abstract, even refers to neural representations. In an answer to the pre-review questions the authors stated:
> Furthermore, we do not have to accept the conclusion that re-training is a necessary part of pruning because a brute force search reveals that neurons can in fact be
> pruned from trained networks in a piecemeal fashion with no retraining and minimal adverse effect on the overall performance of the network. This would be
> impossible if neurons did not belong to the distinct classes we describe."
But this can already be concluded from the 2nd order method, which has a similar characteristic and is based on other 2nd order methods (not shown here). What is the motivation to introduce a new 2nd order method here?
In addition, some other minor conclusions about representations - in particular the cancellation effect - might be based on side-effects of the greedy serial pruning method. Optimally, one would need to consider all the different ways of pruning (which, of course, scales exponentially with the number of neurons and is computationally infeasible). Notably, the authors do consider this limitation in the context of conventional pruning methods in the conclusions: "Third, we assumed that pruning could be done in a serial fashion [...]. We found that all of these assumptions are deeply flawed in the sense that the true relevance of a neuron can only be partially approximated [...] at certain stages of the pruning process". But the brute-force pruning process is also serial - why is that not a problem?
All in all it is unclear to me what the paper adds: there are little conclusions regarding the learnt representations nor is there sufficient benchmarking against state-of-the-art pruning methods. I would suggest to focus the paper in the following way: first, use a state-of-the-art pruning method from the literature (that works without re-training) or do not use any other pruning methods besides brute-force (depending on whether you want to compare pruning methods against brute-force, or want to learn something about the learnt representations). In this way you need to write little about this second-order tuning methods, and readers are not so easily confused about the purpose of this paper (plus it will be considerably shorter!). Then concentrate on 2-layer MNIST and a deeper CIFAR10 network. Further focus the paper by adding an itemised list of the exact contributions that you make, and streamline the paper accordingly. These measures could strongly boost the impact of your work but will require a major revision.
PS: I think the confusion starts with the following sentence in the abstract: "In this work we set out to test several long-held hypothesis about neural network learning representations and numerical approaches to pruning." Both aspects are pretty orthogonal, but are completely mixed up in the paper.


HFH: HOMOLOGICALLY FUNCTIONAL HASHING FOR COMPRESSING DEEP NEURAL NETWORKS
As the complexity of deep neural networks (DNNs) trends to grow to absorb the increasing sizes of data, memory and energy consumption has been receiving more and more attentions for industrial applications, especially on mobile devices. This paper presents a novel structure based on homologically functional hashing to compress DNNs, shortly named as HFH. For each weight entry in a deep net, HFH uses multiple low-cost hash functions to fetch values in a compression space, and then employs a small reconstruction network to recover that entry. The compression space is homological because all layers fetch hashed values from it. The reconstruction network is plugged into the whole network and trained jointly. On several benchmark datasets, HFH demonstrates high compression ratios with little loss on prediction accuracy. Particularly, HFH includes the recently proposed HashedNets (Chen et al., 2015a) as a degenerated scenario and shows significantly improved performance. Moreover, the homological hashing essence facilitates us to efficiently figure out one single desired compression ratio, instead of exhaustive searching throughout a combinatory space configured by all layers.
As the complexity of deep neural networks (DNNs) trends to grow to absorb the increasing sizes of data, memory and energy consumption has been receiving more and more attentions for industrial applications, especially on mobile devices. This paper presents a novel structure based on homologically functional hashing to compress DNNs, shortly named as HFH. For each weight entry in a deep net, HFH uses multiple low-cost hash functions to fetch values in a compression space, and then employs a small reconstruction network to recover that entry. The compression space is homological because all layers fetch hashed values from it. The reconstruction network is plugged into the whole network and trained jointly. On several benchmark datasets, HFH demonstrates high compression ratios with little loss on prediction accuracy. Particularly, HFH includes the recently proposed HashedNets (Chen et al., 2015a) as a degenerated scenario and shows significantly improved performance. Moreover, the homological hashing essence facilitates us to efficiently figure out one single desired compression ratio, instead of exhaustive searching throughout a combinatory space configured by all layers.
1 INTRODUCTION
Deep Neural networks (DNNs) have been receiving ubiquitous success in wide applications, ranging from computer vision (Krizhevsky et al., 2012), to speech recognition (Hinton et al., 2012), natural language processing (Collobert et al., 2011), and domain adaptation (Glorot et al., 2011). As the sizes of data mount up, people usually have to increase the number of parameters in DNNs so as to absorb the vast volume of supervision. High performance computing techniques are studied to speed up training, concerning optimization algorithms, parallel synchronisations on clusters w/o GPUs, and stochastic binarization/ternarization, etc (Courbariaux et al., 2015; Dettmers, 2016; Lin et al., 2016).
On the other hand the memory and energy consumption is usually, if not always, constrained in industrial applications (Kim et al., 2016; Yang et al., 2015). For instance, for commercial search engines (e.g., Google and Baidu) and recommendation systems (e.g., NetFlix and YouTube), the ratio between the increased model size and the improved performance should be considered given limited online resources. Compressing the model size becomes more important for applications on mobile and embedded devices (Han et al., 2016; Kim et al., 2016). Having DNNs running on mobile apps owns many great features such as better privacy, less network bandwidth and real time processing. However, the energy consumption of battery-constrained mobile devices is usually dominated by memory access, which would be greatly saved if a DNN model can fit in on-chip storage rather than DRAM storage (c.f. Han et al. (2016) for details).
A recent trend of studies are thus motivated to focus on compressing the size of DNNs while mostly keeping their predictive performance (Han et al., 2016; Kim et al., 2016; Yang et al., 2015). With different intuitions, there are mainly two types of DNN compression methods, which could be used in conjunction for better parameter savings. The first type tries to revise the training target into more informative supervision using dark knowledge. In specific, Hinton et al. (2014) suggested to train a large network ahead, and distill a much smaller model on a combination of the original labels and the soft-output by the large net. The second type observes the redundancy existence in network weights
(Chen et al., 2015a; Denil et al., 2013), and exploits techniques to constrain or reduce the number of free-parameters in DNNs during learning. This paper focuses on the latter type.
To constrain the network redundancy, several efforts (Denil et al., 2013; Le et al., 2013; Yang et al., 2015) formulated an original weight matrix into either low-rank or fast-food decompositions. Moreover Han et al. (2015) and Han et al. (2016) proposed a simple-yet-effective pruning-retraining iteration during training, followed by quantization and fine-tuning. An adaptive pruning strategy was further studied recently in (Guo et al., 2016) under the name of dynamic network surgery. Chen et al. (2015a) proposed HashedNets to efficiently implement parameter sharing prior to learning, and showed notable compression with much less loss of accuracy than low-rank decomposition. More precisely, prior to training, a hash function is used to randomly group (virtual) weights into a small number of buckets, so that all weights mapped into one hash bucket directly share a same value. HashedNets was further deliberated in frequency domain for compressing convolutional NNs in Chen et al. (2015b). It should be noted that, these approaches are not completely conflicted, and methods based on different assumptions may be used jointly for a better compression.
To obtain a desired compression in applications, one key question of most existing compression approaches is how to appropriately choose the compression ratios. Usually, people have to search within finite choices either heuristically or in a Bayesian manner. The searching procedure becomes more and more exhaustive as the network depth increases, since usually compression ratios are various for different (kinds of) layers (Han et al., 2015; Guo et al., 2016). Moreover, we observe HashedNets compresses model sizes greatly at marginal loss of accuracy for many situations, whereas significantly loses accuracy for others. After revisiting its mechanism, we conjecture this instability comes from at least three factors. First, hashing and training are disjoint in a two-phase manner, i.e., once inappropriate collisions exist, there may be no much optimization room left for training. Second, one single hash function is used to fetch a single value in the compression space, whose collision risk is larger than multiple hashes (Broder & Karlin, 1990). Third, parameter sharing within a buckets implicitly uses identity mapping from the hashed value to the virtual entry in HashedNets.
This paper proposes the homologically functional hashing (HFH) to relieve these problems. Specifically, we use multiple hash functions to map per virtual entry into multiple values in the compression space. Inspired from the “homology” in biology (Williams & Forey, 2004), the compression space is a homological space, meaning that it is accessed by all entries from all (kinds of) layers in a DNN. Then an additional network plays in a mapping function role from these hashed values to the virtual entry (before hashing), which can be also regarded as “reconstructing” the virtual entry from its multiple hashed values. Plugged into and jointly trained within the original network, the reconstruction network is of a comparably ignorable size, i.e., at low memory cost.
The homologically functional hashing structure includes HashedNets as a degenerated special case, and facilitates less value collisions and better value reconstruction. Moreover, the homology essence favors high efficiency during searching appropriate compression ratios, because the only value we have to determine is a scalar, i.e., the size of the homological compression space. Since HFH imposes no restriction on other network design choices (e.g. dropout and weight sparsification), it can be considered as a standard tool for DNN compression, and provably used jointly with other compression techniques for a even better compression. Experiments on several datasets demonstrate significantly larger reduction of model sizes and/or less loss on prediction accuracy, compared with HashedNets.
2 BACKGROUND
Notations. Throughout this paper we express scalars in regular (A or b), vectors in bold (x), and matrices in capital bold (X). Furthermore, we use xi to represent the i-th dimension of vector x, and use Xij to represent the (i, j)-th entry of matrix X. Occasionally, [x]i is also used to represent the i-th dimension of vector x for specification clarity . Notation E[·] stands for the expectation operator.
Feed Forward Neural Networks. We define the forward propagation of the `-th layer as
a`+1i = f(z `+1 i ), with z `+1 i = b `+1 i + d`∑ j=1 V `ija ` j , for ∀i ∈ [1, d`+1]. (1)
For each `-th layer, d` is the output dimensionality, b` is the bias vector, and V` is the (virtual) weight matrix in the `-th layer. Vectors z`,a` ∈ Rd` denote the units before and after the activation
function f(·). Typical choices of f(·) include rectified linear unit (ReLU) (Nair & Hinton, 2010), sigmoid and tanh (Bishop, 1995).
Feature Hashing has been studied as a dimension reduction method for reducing model storage size without maintaining the mapping matrices like random projection (Shi et al., 2009; Weinberger et al., 2009). Briefly, it maps an input vector x ∈ Rn to a much smaller feature space via φ : Rn → RK with K n. Following the definition in Weinberger et al. (2009), the mapping φ is a composite of two approximate uniform hash functions h : N→ {1, . . . ,K} and ξ : N→ {−1,+1}. The j-th element of φ(x) is defined as:
[φ(x)]j = ∑
i:h(i)=j
ξ(i)xi. (2)
As shown in (Weinberger et al., 2009), a key property is its inner product preservation, which we quote and restate below.
Lemma [Inner Product Preservation of Original Feature Hashing] With the hash defined by Eq. (2), the hash kernel is unbiased, i.e., Eφ[φ(x)>φ(y)] = x>y. Moreover, the variance is varx,y = 1 K ∑ i 6=j ( x2i y 2 j + xiyixjyj ) , and thus varx,y = O( 1K ) if ||x||2 = ||y||2 = const.
HashedNets in (Chen et al., 2015a). As illustrated in Figure 1(a), HashedNets randomly maps network weights into a smaller number of groups prior to learning, and the weights in a same group share a same value thereafter. A naive implementation could be trivially achieved by maintaining a secondary matrix that records the group assignment, at the expense of additional memory cost however. HashedNets instead adopts a hash function that requires no storage cost with the model. Assume there is a finite memory budge K` per layer to represent V`, with K` (d` + 1)d`+1. We only need to store a weight vector w` ∈ RK` , and assign V `ij an element in w` indexed by a hash function h`(i, j), namely
V `ij = ξ `(i, j) · w`h`(i,j), (3)
where hash function h`(i, j) outputs an integer within [1,K`]. Another independent hash function ξ`(i, j) : (d`+1 × d`)→ ±1 outputs a sign factor, aiming to reduce the bias due to hash collisions (Weinberger et al., 2009). The resulting matrix V` is virtual, since d` could be increased without increasing the actual number of parameters in w` once the compression space size K` is determined and fixed.
Substituting Eq. (3) into Eq. (1), we have z`+1i = b `+1 i + ∑d` j=1 ξ `(i, j)w`h`(i,j)a ` j . During training, w` is updated by back propagating the gradient via z`+1 (and the virtual V`). Besides, the activation function f(·) in Eq. (1) was kept as ReLU in (Chen et al., 2015a) to further relieve the hash collision effect through a sparse feature space. In both (Chen et al., 2015a) and this paper, the open source xxHash1 is adopted as an approximately uniform hash implementation with low cost.
3 HOMOLOGICALLY FUNCTIONAL HASHING
3.1 STRUCTURE FORMULATION
The key differences between our homologically functional hashing and HashedNets (Chen et al., 2015a) are three folded: (i) how to maintain the compression space, (ii) how to employ hash functions, and (iii) how to map from hashed values to a virtual entry. Precisely,
• Reminder in HashedNets the compression space are split into pieces, with one piece w` for one layer. In contrast, the compression space w ∈ RK in HFH is taken as a whole, so that any entry in all layers fetches hashed values from a same space, namely the homology.
• Instead of adopting one pair of hash function (h, ξ) in Eq. (3), we use a set of multiple pairs of independent random hash functions. Let’s say there are U pairs of mappings {hu, ξu}Uu=1, each hu(i, j) outputs an integer within [1,K], and each ξu(i, j) selects a sign factor.
1http://cyan4973.github.io/xxHash/
• Eq. (3) of HashedNets assigns an equality between one virtual element and one hashed value, i.e., an implicit identity mapping. In contrast, this paper uses a multivariate function g(·) to map from multiple hashed values {ξu(i, j)whu(i,j)}Uu=1 to Vij . Specifically,
Vij = g ([ ξ1(i, j)wh1(i,j), . . . , ξU (i, j)whU (i,j) ] ; α ) . (4)
Therein, α is referred to as the parameters in g(·). Note that the input ξu(i, j)whu(i,j) is order sensitive from u = 1 to U . For optimization convenience, we choose g(·) to be a multi-layer feed forward neural network, and other multivariate functions can be considered as alternatives.
Figure 1(b) illustrates our HFH structure on one layer, which can be easily plugged in any matrices of DNNs. Note that α in the reconstruction network g(·) is of a much smaller size compared to w. For instance, a setting with U = 4 and a 1-layer g(·;α) of α ∈ R4 performs already well enough in experiments. In other words, Eq. (4) just uses an ignorable amount of additional memory to describe a functional w-to-V mapping, whose properties will be further explained in the sequel. Figure 1(c) further shows the HFH structure on multiple layers, taking AlexNet (Krizhevsky et al., 2012) for instance. Therein, no matter convolutional or fully-connected, all layers fetch hashed values from one single homological compression space.
3.2 TRAINING PROCEDURE
The parameters in need of updating include w in the compression and α in g(·). Training HFH is equivalent to training a standard neural network, except that we need to forward/backward-propagate values related to w through g(·) and the virtual entries.
Forward Propagation. Substituting Eq. (4) into Eq. (1), we omit the super-script ` and get
zi = bi + d∑ j=1 ajVij = bi + d∑ j=1 aj · g ([ ξ1(i, j)wh1(i,j), . . . , ξU (i, j)whU (i,j) ] ; α ) . (5)
Backward Propagation. Denote L as the final loss function, e.g., cross entropy or squared loss, and suppose δi = ∂L∂zi is already available by back-propagation from layers above. The derivatives of L with respect to w and α are computed by
∂L ∂w = ∑ i ∑ j ajδi ∂Vij ∂w , ∂L ∂α = ∑ i ∑ j ajδi ∂Vij ∂α , (6)
where, since we choose g(·) as a multilayer neural network, derivatives ∂Vij∂w and ∂Vij ∂α can be calculated through the small network g(·) in a standard back-propagation manner.
Complexity. Concerning time and memory cost, HFH roughly has the same complexity as HashedNets, since the additional small network g(·) is quite light-weighted (e.g., 4 hashed inputs and 2 layers). One key varying factor is the way to implement multiple hash functions. On one hand, if they are calculated online, HFH requires little additional time if tackling them in parallel. On the other, if they are pre-computed and stored in dicts to avoid hashing time cost, the multiple hash functions of HFH demand more storage space. In application, we suggest to pre-compute hashes during offline training for speedup, and to compute hashing in parallel during online prediction for saving memory under limited hardware budget.
3.3 PROPERTY ANALYSIS
In this part, we try to depict the properties of our HFH from several aspects to help understanding it, especially in comparison with HashedNets (Chen et al., 2015a).
Value Collision. It should be noted, both HashedNets and HFH conduct hashing prior to training, i.e., in a two-phase manner. Consequently, it would be unsatisfactory if hashing collisions happen among important values. For instance in natural language processing tasks, one may observe wired results if there are many hashing collisions among embeddings (which form a matrix) of frequent words, especially when they are not related at all. In the literature, multiple hash functions are known to perform better than one single function (Azar et al., 1994; Broder & Karlin, 1990; Broder & Mitzenmacher, 2001). In intuition, when we have multiple hash functions, the items colliding in one function are hashed differently by other hash functions.
Value Reconstruction. In both HashedNets and HFH, the hashing trick can be viewed as a reconstruction of the original parameter V from w ∈ RK . In this sense, the approach with a lower reconstruction error is preferred2. Then we have at least the following two observations:
• The maximum number of possible distinct values output by hashing intuitively explains the modelling capability (Shi et al., 2009). For HashedNets, considering the sign hashing function ξ(·), we have at most 2K possible distinct values of Eq. (3) to represent elements in V. In contrast, since there are multiple ordered hashed inputs, HFH has at most (2K)U possible distinct values of Eq. (4). Note that the memory size K is the same for both.
• The reconstruction error may be difficult to analyzed directly, since the hashing mechanism is trained jointly within the whole network. However, we observe g ([ ξ1(i, j)wh1(i,j), . . . , ξU (i, j)whU (i,j) ] ; α )
degenerates to g(ξ1(i, j)wh1(i,j)) if we assign zeros to all entries in α unrelated to the 1st input dimension. Since g(ξ1(i, j)wh1(i,j)) depends only on one single pair of hash functions, it is conceptually equivalent to HashedNets. Consequently, including HashedNets as a special case, HFH with freely adjustable α is able to reach a lower reconstruction error to fit the final accuracy better.
2One might argue that there exists redundancy in V, whereas here we could imagine V is already structured and filled by values with least redundancy.
Feature Hashing. In line with previous work (Shi et al., 2009; Weinberger et al., 2009), we compare HashedNets and HFH in terms of feature hashing. For specification clarity, we drop the sign hashing functions ξ(·) below for both methods, the analysis with which is straightforward by replacing K hereafter with 2K.
• For HashedNets, one first defines a hash mapping function φ(1)i (a), whose k-th element is[ φ
(1) i (a) ] k , ∑ j:h(i,j)=k aj , for k = 1, . . . ,K. (7)
Thus zi by HashedNets can be computed as the inner product (details c.f. Section 4.3 in (Chen et al., 2015a))
zi = w >φ (1) i (a). (8)
• For HFH, we first define a hash mapping function φ(2)i (a). Different from a K-dim output in Eq. (7), it is of a much larger size KU , with (∑U u=1 kuK (u−1) )
-th element as[ φ
(2) i (a) ]∑U u=1 kuK (u−1) , ∑ j:h1(i,j)=k1 h2(i,j)=k2
... hU (i,j)=kU
aj , for ∀u, ku = 1, . . . ,K. (9)
Second, we define vector gα(w) still of length KU , whose (∑U u=1 kuK (u−1) ) -th entry is
[gα(w)]∑U u=1 kuK (u−1) , g (wk1 , wk2 , . . . , wkU ; α) , for ∀u, ku = 1, . . . ,K. (10)
Thus zi by HFH can be computed as the following inner product
zi = gα(w) > φ (2) i (a). (11)
The difference between Eq. (8) and Eq. (11) further explains the above discussion about “the maximum number of possible distinct values”.
Homology. Efficiency during searching appropriate compression ratios is the most important benefit brought by the homological property. Particularly, people usually have to search within candidate compression ratios either heuristically or in a Bayesian manner. The searching procedure becomes more and more exhaustive as the network depth increases, since usually compression ratios are various for different (kinds of) layers. For instance, the compression ratios of convolutional and fully-connected layers in both Han et al. (2015) and Guo et al. (2016) are quite different. In contrast for HFH, we only need to vary one scalar value K concerning the compression ratio. One may argue that these methods can adopt a same pruning ratio throughout layers, while preliminary experiments (not focused on and thus omitted in this paper) indicate equally pruning leads to inferior performances.
4 RELATED WORK
Recent studies have confirmed the redundancy existence in the parameters of deep neural networks. Denil et al. (2013) decomposed a matrix in a fully-connected layers as the product of two low-rank matrices, so that the number of parameters decreases linearly as the latent dimensionality decreases. More structured decompositions Fastfood (Le et al., 2013) and Deep Fried (Yang et al., 2015) were proposed not only to reduce the number of parameters, but also to speed up matrix multiplications. Han et al. (2015) and Han et al. (2016) proposed to iterate pruning-retraining during training DNNs, and used quantization and fine-tuning as a post-processing step. Huffman coding and hardware implementation were also considered. In order to mostly keep accuracy, the authors suggested multiple rounds of pruning-retraining. That is, for little accuracy loss, we have to prune slowly enough and thus suffer from increased training time. More recently, dynamic network surgery was proposed in (Guo et al., 2016), which achieves state-of-the-art compressions by adopting a dynamic strategy to adapt the pruning procedure. Again, the most related work to ours is HashedNets (Chen et al., 2015a), which was then extended in (Chen et al., 2015b) to random hashing in frequency domain for compressing convolutional neural networks. Either HashedNets or HFH could be combined in conjunction with other techniques for better compression.
Extensive studies have been made on constructing and analyzing multiple hash functions, which have shown better performances over one single hash function (Broder & Karlin, 1990). One multi-hashing algorithm, d-random scheme (Azar et al., 1994), uses only one hash table but d hash functions, pretty similar to our settings. One choice alternative to d-random is the d-left algorithm proposed in Broder & Mitzenmacher (2001), used for improving IP lookups. Hashing algorithms for natural language processing are also studied in Goyal et al. (2012). Papers (Shi et al., 2009; Weinberger et al., 2009) investigated feature hashing (a.k.a. the hashing trick), providing useful bounds and feasible results.
5 EXPERIMENTS
We conduct extensive experiments to evaluate HFH on DNN compression, especially in comparison with HashedNets. Codes for fully reproducibility will be open source in future after necessary polishment.
5.1 ENVIRONMENT DESCRIPTIONS
Datasets. Five benchmark datasets (Larochelle et al., 2007) are considered here, including (1) the original MNIST hand-written digit dataset, (2) dataset BG-IMG as a variant to MNIST, (3) binary image classification dataset CONVEX, (4) dataset CIFAR-10, and (5) ImageNet ILSVRC-2012. For all datasets, we use prespecified training and testing splits. In particular, the original MNIST dataset has #train=60,000 and #test=10,000, while both BG-IMG and CONVEX have #train=12,000 and #test=50,000. CIFAR-10 consists of 60k 32×32 color images of 10 object classes, i.e., 6k images per class, and there are 50k training and 10k testing images, respectively. ImageNet ILSVRC-2012 has 1.2M training images and 50k validation images. Moreover, collected from a commercial search engine, a large scale dataset with billions of samples is used to learn DNNs for pairwise semantic ranking. We randomly split out 20% samples from the training data to form the validation set.
Methods and Settings. Chen et al. (2015a) compared HashedNets against several DNN compression approaches, and showed HashedNets performs consistently the best, including the low-rank decomposition (Denil et al., 2013). Under the same settings, we compare HFH with HashedNets3 and baseline DNNs without compression. If not defined, all activation functions are chosen as ReLU.
5.2 FEEDFORWARD NNS FOR CLASSIFICATION
Focusing on classification by feedforward NNs, the performances of HFH are tested on MNIST, textttBG-IMG and CONVEX in the following two scenarios. First, we compare the effects of varying compression ratio by HFH and HashedNets. Second, we compare different configurations of HFH itself, including the number U of seeds and the layer of reconstruction network g(·). Hidden layers within g(·) keep using tanh as activation functions.
5.2.1 VARYING COMPRESSION RATIO
To test robustness, we vary the compression ratio with (1) a fixed virtual network size, and then with (2) a fixed memory size (i.e., the size of compression space). Three-layer (1 hidden layer) and five-layer (3 hidden layers) networks are investigated. In experiments, we vary the compression ratio geometrically within {1, 12 , 1 4 , . . . , 1 64}. For HFH, this comparison sticked to use 4 hash functions, 3-layer g(·), and without dual space hashing.
With Virtual Network Size Fixed. The hidden layer for 3-layer nets initializes at 1000 units, and for 5-layer nets starts at 100 units per layer. As the compression ratio ranges from 1 to 1/64 with a fixed virtual network size, the memory decreases and it becomes increasingly difficult to preserve the classification accuracy. The testing errors are shown in Figure 2, where standard neural networks with equivalent parameter sizes are included in comparison. HFH shows robustly effective compression against the compression ratios, and persistently produces better prediction accuracy than HashedNets. It should be noted, even when the compression ratio equals to one, HFH with the reconstruction network structure is still not equivalent to HashedNets and performs better.
3HashedNets code downloaded from http://www.cse.wustl.edu/∼wenlinchen/project/HashedNets/index.html
With Memory Storage Fixed. We change to vary the compression ratio from 1 to 1/64 with a fixed memory storage size, i.e., the size of the virtual network increases while the number of free parameters remains unchanged. In this sense, we’d better call it expansion instead of compression. Both 3-layer and 5-layer nets initialize at 50 units per hidden layer. The testing errors in this scenario are shown in Figure 3. At all compression (expansion) ratios on each dataset, HFH performs better than or at least comparably well compared to HashedNets.
5.2.2 VARYING CONFIGURATIONS OF HFH
On 3-layer nets with compression ratio 1/8, we vary the configuration dimensions of HFH, including the number of hash functions (U ), and the structure of layers of the reconstruction network g(·). Since it is impossible to enumerate all probable choices, U is restricted to vary in {2, 4, 8, 16}. The structure of g(·) is chosen from 2 ∼ 4 layers, with U ×1, U ×0.5U ×1, U ×U ×0.5U ×1 layerwise widths, respectively. We denote Ux-Gy as x hash functions and y layers of g(·). Table 1 shows the performances of HFH with different configurations on MNIST.
We summarize key observations below. First, the series from index (0) to (1.x) fixes a 3-layer g(·) and varies the number of hash functions. As listed, more hash functions do not ensure a better accuracy, and instead U4-G3 performs the best, perhaps because too many hash functions potentially brings too many partial collisions. Second, the series from (0) to (2.x) fixes the number of hash functions and varies the layer number in g(·), where three layers performs the best mainly due to its strongest representability.
5.3 CONVOLUTIONAL NNS FOR IMAGE CLASSIFICATION
We further evaluate the performances by hashing compression methods on deep convolutional NNs for image classification. Note that, we treat the convolutional layers same as fully-connected layers during hashing. A more meticulous alternative could be hashing over CNNs on CIFAR-10 (Krizhevsky & Hinton, 2009) and ImageNet ILSVRC-2012 datasets are reported here for comparison.
Particularly on CIFAR-10, a traditional CNN model with 3 conv layers and 2 fc layers is trained as the baseline, which stores roughly 4.8 million parameters and achieves 11.01% top-1 classification test error. As we vary the compression ratio, again equally over all layers, the top-1 test errors by HashedNets and HFH are illustrated in Figure 4. As shown, HFH can still achieve a test error at 18% with just 1 percent memory size of the baseline model, much better than HashedNets.
On ImageNet ILSVRC-2012 data, the standard AlexNet model (Krizhevsky et al., 2012) is chosen as the reference model, which has 61 million parameters across 5 conv layers and 3 fc layers. The top1 classification test error by this baseline AlexNet is 42.78%. The current state-of-the-art compression results on AlexNet are achieved by pruning in (Han et al., 2015) with overall 1/9 compression ratio,
and dynamic network surgery in (Guo et al., 2016) with overall 1/17.7 compression ratio, which are obtained by carefully tuning the compression ratios in different layers.
Applying HashedNets to match the above two compression ratios, the performances are much worse, as listed in Table 2. However, using HFH at the same compression ratios, the test errors are significantly reduced and even comparable to those by (Han et al., 2015) and (Guo et al., 2016). Again it is noticeable that: (1) both HashedNets and HFH hold the compression ratios across all layers equal, and thus the searching time cost is low; (2) the hashing techniques and the pruning strategies are not mutually exclusive, but provably to be used jointly for a better compression.
5.4 CONVOLUTIONAL NNS FOR PAIRWISE SEMANTIC RANKING
Finally, we evaluate the performance of HFH on semantic learning-to-rank convolutional NNs. The data are collected from logs of a commercial search engine, with per clicked query-url being a positive sample and per non-clicked being a negative sample. There are totally around 45 billion samples. We adopt a deep convolutional structured semantic model similar to (Huang et al., 2013; Shen et al., 2014), which is of a siamese structure to describe the semantic similarity between a query and the title of an url. Stochastically for each query, the network is trained to optimize the pairwise ranking cross entropy for each randomly sampled pair of positive and negative samples.
The performance is evaluated by correct-to-wrong pairwise ranking ratio on testing set. In Figure 5, we plot the performance by a baseline network as training epoch proceeds. Results by HFH and HashedNets with 1/4 compression ratio are included for comparison. Still, HFH performs better than HashedNets throughout the training epochs, and even comparable to the full network baseline that requires 4 times of memory storage. The deterioration of HashedNets probably comes from many inappropriate collisions on word embeddings, especially for words of high frequencies.
6 CONCLUSION AND FUTURE WORK
This paper presents a novel homologically functional hashing for neural network compression. Briefly, after adopting multiple low-cost hash functions to fetch values in a homological compression space, HFH employs a small reconstruction network to recover each entry in an matrix of the original network. The compression space is shared throughout all layers, and the reconstruction network is plugged into the whole network and learned jointly. HFH includes the recently proposed HashedNets (Chen et al., 2015a) as a degenerated special case. On multiple datasets, HFH demonstrates promisingly high compression ratios with little loss on prediction accuracy, especially significantly improved compared with HashedNets. As a simple and effective approach, HFH is expected to be a standard tool for DNN compression.
As future work, we aim for deeper analysis on the properties and bounds of HFH. More industrial applications are also expected, especially on mobile devices. Furthermore, HFH might be extended in a multi-hops hyper-structure. Briefly, imagining w in the compression space plays a virtual role similar to V, we may adopt the homologically functional hashing structure once again on w, so that w is still virtual and reconstructed by hashed values from another further compressed space.

The paper describes an extension of the HasheNets work, with several novel twists. Instead of using a single hash function, the proposed HFH approach uses multiple hash function to associate each "virtual" (to-be-synthesized) weight location to several components of an underlying parameter vector (shared across all layers). These components are then passed through a small MLP to synthesize the final weight.
This is an interesting and novel idea, and the experiments demonstrate that it improves substantially over HashedNets. However, HashedNets is not a particularly compelling technique for neural network model compression, especially when compared with more recent work on pruning- and quantization-based approaches. The experiments in this paper demonstrate that the proposed approach yields worse accuracy at the same compression ratios as pruning-based approaches, while providing no runtime speedup benefits. While the authors mention the technique is only 20% slower (which I am pleasantly surprised by), I don't understand why this technique should ever be used over competing approaches for the kinds of networks the authors present experimental results on. The authors suggest that the technique could be combined with pruning based approaches... this may be true, but no experiments to this effect are provided. The paper also suggests that ease of setting the compression ratio is a benefit of HFH, but I don't think that's a sufficient win to justify the numerous other downsides (in accuracy and speed).
In response to a question, the authors point out that the technique works very well for compressing embeddings, and for this setting the technique does appear like a genuinely useful contribution, given the marginal overhead and substantial train-time benefits. If the paper focused on this setting and showed experimental results on e.g. language modeling tasks or other scenarios with high-dimensional sparse/one-hot inputs require large embedding layers, I could enthusiastically recommend acceptance. However for the CNN and MLP networks which are the main focus of the experiments, I don't think the technique is suitable, as much as I like the basic idea.