Magnitude-based pruning is one of the simplest methods for pruning neural networks. Despite its simplicity, magnitude-based pruning and its variants demonstrated remarkable performances for pruning modern architectures. Based on the observation that the magnitude-based pruning indeed minimizes the Frobenius distortion of a linear operator corresponding to a single layer, we develop a simple pruning method, coined lookahead pruning, by extending the single layer optimization to a multi-layer optimization. Our experimental results demonstrate that the proposed method consistently outperforms the magnitude pruning on various networks including VGG and ResNet, particularly in the high-sparsity regime.
*Summary*
The paper proposes a multi-layer alternative to magnitude-based pruning. The operations entailed in the previous, current, and subsequent layers are treated as linear operations (by omitting any nonlinearities), weights are selected for pruning to minimize the "Frobenius distortion", the Frobenius norm of the difference between products of the (i-1, i, i+1)-layer Jacobians with and without the selected weight. This simplifies to a cost-effective pruning criterion. In spite of the simplistic linear setting assumed for the derivation, results show the criterion prunes better than weight-based methods at unstructured pruning of a variety of modern architectures with CIFAR-10, particularly excelling at higher sparsity.
*Rating*
The paper has some clear positives, particularly:
    + Clear writing and formatting
    + Simple method
    + Good structure for the experimental analysis (with 5x replications!)
However there are a few limitations, noted below; while none is fatal on its own, in total the limitations have led me to recommend "weak reject" currently.
Limitations of the method:
(1) Residual networks: The lack of an explicit strategy for handling residual connections (and the accompanying worsened relative performance) is a notable limitation since residual/skip connections are nearly universal in state-of-the-art large networks. The performance was shown to still be *slightly* better than with magnitude pruning.
(2) Global ranking: Since connections are pruned layerwise, rather than taking the best-k neurons across the entire network at once, I assume that the LAP pruning criterion doesn't scale reasonably across layers. This implies that the method cannot be used to learn network structure. Instead the user must decide the desired number of neurons at each layer.
(3) Structured pruning: There is no mention of pruning entire convolutional kernels or "neurons" at once, so I assume that only individual weights were pruned. Since structured pruning is the simplest way to achieve speedup in network inference (as opposed to merely reduction in model size), how does the LAP criterion perform when adapted for structured pruning, e.g. removing filters/neurons with the best average LAP score?
Limitations of the experiments:
(4) Baselines: While the paper is explicitly focused on an easy to compute replacement for magnitude-based pruning, there are a wide variety of alternative methods available. These vary in complexity, runtime, etc., but they deserve mention and either explicit comparison in the experiments or reasoning to justify the omission of such comparisons.
(5) ImageNet: (Insert obligatory statement about the ubiquity of ImageNet experiments, ...) While it is cliche to request ImageNet experiments and CIFAR-10 is a helpful stand-in, they would be really nice to have.
(6) Activations after non-linearities: While Fig. 3 and the remaining experiments present a reasonable case that the presence of non-linearities doesn't prevent LAP from improving upon magnitude-based pruning, it doesn't resolve the issue either. Whether considering negative values clipped by with ReLU or large magnitude values that are squashed by sigmoid and tanh, the linear-only model is a poor approximation for some unknown fraction of neurons for probably most inputs. Does this mean that LAP is underperforming in those cases? Are those cases sufficiently rare or randomly distributed that they are merely noise? Is there another mechanism at play? In practical terms, how much does the activation rate (positive for ReLU, linear/unsquashed for sigmoid/tanh) vary by neuron? This seems like a reasonably simple thing to compute and incorporate into pruning.
*Notes*
Eq. (4): Is (4) simply the one-step/greedy approximation to the optimization in (3)? If so, it may be helpful to state this explicitly. Also, is $w = W_i[j,k]$? If so, this is useful to explicitly state.
Sec 2.1: Consider noting that the linear-model setup is used to construct the method, but non-linearities are addressed subsequently
Sec 2.2: Is the activation probability p_j used in practice, or is it merely an explanatory device?
pg5: "gradually prune (p/5)%" and marked with a suffix '-seq'"
pg5: note that residual connections are discussed in the experiments?
Tables 3-6: note that these all use CIFAR-10