Pruning units in a deep network can help speed up inference and training as well as reduce the size of the model. We show that bias propagation is a pruning technique which consistently outperforms the common approach of merely removing units, regardless of the architecture and the dataset. We also show how a simple adaptation of an existing scoring function allows us to select the best units to prune. Finally, we show that the units selected by the best performing scoring functions are somewhat consistent over the course of training, implying the dead parts of the network appear during the early stages of training.
This paper proposes a simple improvement to methods for unit pruning. After identifying a unit to remove (selected by the experimenter’s pruning heuristic of choice), the activation of that unit is approximately incorporated into the subsequent unit by “mean replacement”. The mean unit activation (computed on a small subset of the training set) is multiplied by each outgoing weight (or convolutional filter) and added to each corresponding bias instead. Experiments show this method is generally better than the typical method of zero-replacement before fine-tuning, though the advantage is smaller after several epochs of fine-tuning.
While I find this paper intriguing and applaud the extensive experimentation and documentation, I have some concerns as well:
	1. There are unanswered questions about how this method relates to existing work. It is not clear from the paper how the “mean replacement” method differs from the two most related works (Ye, 2018) and (Morcos, 2018), which propose variations on replacing units with constant values or mean activations, respectively. Also, why does the method in this paper seem to yield good results, while the related method (Morcos, 2018) yields “inferior performance”?
	2. The results are stated to only apply to networks “without batch normalization”. The reason seems intuitive: any change that can be merely rolled into the bias will be lost after normalization (depending perhaps on the ordering of normalization and the non-linearities). This leaves an annually decreasing fraction of networks to which this method is applicable, given the widespread use of batch norm.
	3. Critically, it’s difficult to compare this work against other pruning works given the lack of results reported in terms of final test error and the lack of the ubiquitous “error vs. %-pruned” plot.
	
Overall, this paper is lacking some clarity, may be limited in originality, may be helpful for some common networks and composable with other pruning methods (significance), but has a good quality evaluation (subject to the clarity issues). I’m rating this paper below the threshold given the limitations, but I’m willing to consider an upgrade to the score if these questions are addressed.
Other notes:
	4. What is your definition of a convolutional “pruning unit”? (From context, I’d presume it corresponds to an output activation map.)
	5. In Section 3.1:  replace “in practice, people …” with  something like “in practice, it is common to”.
	6. In Equation 3, is the absolute value of the pruning penalty used in the evaluation?
	7. In the footnote in Section 3.2, how many training samples are needed for a good approximation? How many are used in the experiments?
	8. There are a couple typos in Section 3.2: “replacing -the- these units with zeroes” and “each of these output*s*”.
	9. Presumably the “\Delta Loss after pruning” in Figures 2-6 is validation or test loss, not training loss? Is this the cross-entropy loss? Also, it would be much easier to compare to other papers if test accuracy were reported instead or in addition.
	10. In Figure 4, the cost to recover using fine-tuning seems to be only roughly 2% of the original training time. How much time is lost to the process of computing the average unit activation?
UPDATE: I've raised the score slightly to 5 after the rebuttals and revisions.