Neural network pruning is a popular technique for reducing inference costs by removing connections, neurons, or other structure from the network. In the literature, pruning typically follows a standard procedure: train the network, remove unwanted structure (pruning), and train the resulting network further to recover accuracy (fine-tuning).
In this paper, we explore an alternative to fine-tuning: rewinding. Rather than continuing to train the resultant pruned network (fine-tuning), rewind the remaining weights to their values from earlier in training, and re-train the resultant network for the remainder of the original training process. We find that this procedure, which repurposes the strategy for finding lottery tickets presented by , makes it possible to prune networks further than is possible with finetuning for a given target accuracy, provided that the weights are rewound to a suitable point in training. We also find that there is a wide range of suitable rewind points that achieve higher accuracy than fine-tuning across all tested networks. Based on these results, we argue that practitioners should explore rewinding as an alternative to fine-tuning for neural network pruning.
*Summary*
Extending the observations of Frankle et al. (2019, "The Lottery Ticket Hypothesis"), this paper examines "rewinding" as an alternative to fine-tuning in a typical network pruning process. After training to convergence for T iterations, the k% of weights with the smallest magnitude are pruned (set to zero). Typically, in fine-tuning, the remaining weights are continued to be trained for several more iterations at a small learning rate. With "rewinding", the remaining weights are reset to their values in iteration 0<=t<T and are trained for T-t iterations with the original learning rate schedule.
Experiments with this method (and an iterative variant) show that (a) rewinding achieves 2-5x smaller networks matching the unpruned accuracy (with comprehensive tuning of the rewinding parameter t); (b) selection of rewinding iteration is important, but is flexible within a large range for higher sparsity pruning.
*Rating*
The paper is very well written with good exposition, thorough notes and citations for all methodological choices,
and an explicit statement of the limitations of the work.
A few considerations relevant to the rating:
(1) Novelty: The idea for rewinding is not novel, as acknowledged clearly in the paper. Frankle et al. (2019, "Stabilizing the Lottery Ticket Hypothesis") showed that rewinding to an early iteration of training (0<t<T) yielded better accuracy than rewinding to iteration t=0 for VGG-19 and ResNet-18 on CIFAR-10. However, Frankle et al. did not consider fine-tuning as it was not relevant to lottery ticket discovery. This submission studies the tradeoffs of rewinding vs. fine-tuning.
(2) Thoroughness: The paper considers ResNet-20 and VGG-16 with CIFAR-10 and ResNet-50 with ImageNet. Conclusions would be strengthened with additional combinations of networks and datasets.
(3) Acknowledged limitations: As noted, the paper doesn't consider any pruning criteria other than weight magnitude, nor does it consider structured pruning. The latter in particular is important for applications where prediction speed on commodity hardware is a limiting factor.
As it is, I think this paper a worthy (if limited) contribution to the understanding of network pruning.
*Notes*
Table 1/Figures *: note which dataset is used for each architecture
Figures 2-3: It seems that many values are clipped by the legend range of +/- 0.5%. Consider showing the figure with a larger range or adding such a figure to the appendix.