Overparameterization has been shown to benefit both the optimization and generalization of neural networks, but large networks are resource hungry at both training and test time. Network pruning can reduce test-time resource requirements, but is typically applied to trained networks and therefore cannot avoid the expensive training process. We aim to prune networks at initialization, thereby saving resources at training time as well. Specifically, we argue that efficient training requires preserving the gradient flow through the network. This leads to a simple but effective pruning criterion we term Gradient Signal Preservation (GraSP). We empirically investigate the effectiveness of the proposed method with extensive experiments on CIFAR-10, CIFAR-100, Tiny-ImageNet and ImageNet, using VGGNet and ResNet architectures. Our method can prune 80% of the weights of a VGG-16 network on ImageNet at initialization, with only a 1.6% drop in top-1 accuracy. Moreover, our method achieves significantly better performance than the baseline at extreme sparsity levels.
This paper proposes a novel one-shot-pruning algorithm which improves the training of sparse networks by maximizing the norm of the gradient at initialization. The utility of training sparse neural networks and shortcomings of dense-to-sparse algorithms like Pruning, LotteryTicket are nicely motivated at introduction. The pruning criterion is motivated by the first order approximation of the change in the gradient norm when a single connection is removed, though the results show that removing many connections together with GraSP increases the total gradient norm therefore allowing the loss to decrease faster. Experiments suggest employing such pruning algorithm improves final performance over two baselines: random and SNIP. 
Though I find the proposed method intriguing and well motivated, experiments section of the paper misses some important sparse training baselines and needs some improvement. I am willing to increase my score given my concerns/questions below are addressed. 
(1) The paper doesn't mention some important prior work on the topic. Since the paper focuses on end-to-end sparse training, the following sparse training methods needs to be considered and compared with:
- Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science [Mocanu, 2018]
- Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization [Mostafa, 2019]
- Deep Rewiring: Training very sparse deep networks [Bellec, 2017] 
- There is also few recent work submitted to ICLR2020: https://openreview.net/forum?id=SJlbGJrtDB, https://openreview.net/forum?id=ryg7vA4tPB, https://openreview.net/forum?id=ByeSYa4KPS
(2) Pruning baselines can be improved. I am not convinced that they represent the best achievable sparse training results. I would recommend method proposed by `Prune or Not to Prune` as a strong baseline. You can also check `State of Sparsity` paper to obtain some competitive pruning results.
(3) It's great that the authors are aware of the importance of having experiments on larger datasets. Though, I found the results reported on Imagenet to be limited. Is there a reason why Imagenet-2012 results are missing pruning baselines? I think having other reported pruning results here along with performance of other sparse training methods (SET, DSR) would be useful. Most of these numbers should be readily available in the papers mentioned above, but I guess it is always better to run them using the same settings. 
(4) To demonstrate the usefulness of the pruning criteria proposed, it would be nice to do some simple ablations. Some suggestions: (1) Remove weights that would *decrease* the gradient norm most (2) Do random pruning while preserving exact per layer sparsity fractions. (3) sweep over batch size used to calculate the importance scores and evaluate final accuracies or the initial gradient norm. The second experiment would help identifying whether the gains are due to better allocation of sparsities across layers or due to increased gradient norm. Looking at Figure-4 and seeing the per layer sparsities are different, It is not clear to me which one is the underlying reason for improved performance.
(5) (Page 8 / Table 5) Do you aggregate all accuracies in Table 5 using different batch sizes and initialization methods? If, so I am not sure what the intended message here is, since it is difficult to infer how these hyper-parameters affect the result. Do you sweep different batch sizes for estimating importance of units, too? It would be nice to see whether the two batch sizes interact with each other and/or how increased batch size affects the quality of pruned networks. 
Some minor comments:
(a) (Page 1) I found the motivation very intriguing. Though the statement `Recently, F&C (2019) shed light on this...` seems a bit off, given that LT can't find solutions as well as the pruning solution in most practical (larger datasets and architectures) settings. Therefore I would be better to pose this as an `open problem`. 
(b) (end of page-1) `However, connection sensitivity is sub-optimal as a criterion because the gradient of each weight might change dramatically after pruning due to complicated interactions between weights`. I think this is still the case for GraSP. Since the criterion it uses assumes independence (i.e. what if we remove a single weight?). It would be nice to see some ablations on this. Does `K=number of weights removed` affect the norm of the sparsified networks?
(c) (Figure 1) I find the comparative illustration between SNIP and GraSP very useful. Though, the architecture presented seems a bit artificial (i.e. I am not aware of any architecture with single hidden layer and a single output unit). I think the same motivation can be made by removing the top unit (therefore having 6-4-1 units) and removing all incoming connections for the output unit until a single connection remains. Then SNIP would remove that single connection whereas GraSP would remove one of the connections in the previous layer.
(d) (Section 2.1) `In contrast, Hessian based algorithms...` Though it is a structured pruning algorithm It might be nice to include the following work, https://arxiv.org/abs/1611.06440. 
(e) (Section 2.1) Previous work needs following citations:  [Bellec, 2017], [Mocanu, 2018] and [Mostafa, 2019] 
(f) (Section 2.2) Why the initial dynamics affect the final performance? One explanation given in the paper is through recent work on NTK and this is great. Though training settings used at `Lee et al (2019a)` and in the paper are a bit different. Usage of MSE, small datasets, etc… So it might be nice to point out differences. 
(g) (Section 3) At $D = {(x_i, y_i)}_{i=1}^n$, `n`->`N` 
(h) (Page 4) `Preserving the loss value motivated several…` -> `motivated by several…`
I think it is better to use existing terminology whenever available.I think using `One-shot pruning` instead of `Foresight pruning` would be a better choice and would prevent confusion. 
(j) (Page 5) `However, it has been observed that different weights are highly coupled …` This has been observed much earlier, too: like in Hassibi, 1993. 
(k) (Page 7) Last sentence `and thus hopefully..`: needs to be fixed.
(l) (Page 8) The whole page needs some proof-reading. Some of them: (a) `SNIP and GraSP. We present...` probably connect with comma (b) `aims for preserving` -> `aims to preserve` (c) `In contrast, SNIP are more` `are`->`is` (d) `for ablation study` -> `as an ablation study`... 
(m) Is there a specific reason why VGG networks are preferred for experiments? I don't think they are relevant to any practical application anymore and they are massively over-parameterized for tasks in hand. Specifically for Cifar-10. I think focusing on more recent networks and larger datasets would increase the impact of the work.