We examine how recently documented, fundamental phenomena in deep learning models subject to pruning are affected by changes in the pruning procedure. Specifically, we analyze differences in the connectivity structure and learning dynamics of pruned models found through a set of common iterative pruning techniques, to address questions of uniqueness of trainable, high-sparsity subnetworks, and their dependence on the chosen pruning method. In convolutional layers, we document the emergence of structure induced by magnitude-based unstructured pruning in conjunction with weight rewinding that resembles the effects of structured pruning. We also show empirical evidence that weight stability can be automatically achieved through apposite pruning techniques.
*Summary*
This paper compares network pruning masks learned via different iterative pruning methods. Experiments on LeNet + MNIST show (a) different methods can achieve similar accuracy, (b) pruned sub-networks may differ significantly despite identical initialization, (c) weight reinitialization between pruning iterations yields more structured convolutional layer pruning than not reinitializing, and (d) pruning methods may differ in the stability of weights over pruning iterations.
*Rating*
There are interesting bits of data in this paper, but the overall story is somewhat muddled and some inferences seem to be insufficiently supported by data (1-2 below). In addition, the text would benefit from better organization and presentation (3-4 below) and replications on other datasets and architectures (5 below). As a result, my rating is currently weak reject.
(1) *Overlap in pruned sub-networks*: In the middle of Sec. 4, Fig 3-5 examine the similarity of pruning masks between methods. It seems clear from several of the plots that multiple methods produce identical layer-wise masks, e.g. Fig 3(a), while others show a wide variance. The overlap in lines makes this difficult to assess at times: perhaps a table would communicate it better? Also, are Fig 3-4 depicting the Jaccard distance between masks of unpruned or pruned weights? Is the ordering of training samples fixed in addition to network initialization? Is reinitialization used between iterations? Also, Fig 5 seems to contradict the conclusion that methods tend to learn different masks, since the structures are noticeably similar.
(2) *Weight stability during pruning*: It is difficult to discern a conclusion in Sec 5. First, a clarification on the figures: are lines for pruned weights terminated where they are pruned? If so, this would be helpful to state. The 4th paragraph claims, "we empirically find a correlation between weight stability and performance", but this is not at all obvious from Figures 6-7. I'm not sure what a more stable evolution looks like. Hybrid is shown to be accurate in Fig 1, but the conv. weights in 6(a) are a spaghetti tangle and the FC weights in 7(a) are constantly increasing in magnitude. Perhaps a mathematical formulation for stability (perhaps based on average standard deviation of each weight's values over training) with a table of values for each method/layer would help to clarify.
(3) *Organization*: Since the paper has many intertwined observations, a better organization would be helpful. Consider mirroring the structure of Sec 1.1 in a combined Sec. 4-5 with clear paragraph headers summarizing each conclusion.
(4) *Presentation*: Figure is too small throughout to read from a printed copy (or even on a screen without significant zooming). Several results could be presented with less ambiguity in tabular form, as noted above.
(5) *Replications*: The paper presents results only a single set of experiments using the MNIST dataset with the LeNet architecture. While this isn't a fatal issue, it is a significant weakness.
*Notes*
Fig 1 and 2: What spacing is used for the x- and y- axes?
Fig 8: Perhaps scale vertically by the standard deviation of the weights?