The recent success of the lottery ticket hypothesis by Frankle & Carbin (2018) suggests that small, sparsified neural networks can be trained as long as the network is initialized properly. Several followup discussions on the initialization of the sparsified model have discovered interesting characteristics such as the necessity of rewinding (Frankle et al. (2019)), importance of sign of the initial weights (Zhou et al. (2019)), and the transferability of the winning lottery tickets (S. Morcos et al. (2019)). In contrast, another essential aspect of the winning ticket, structure of the sparsified model, has been little discussed. To find the lottery ticket, unfortunately, all the prior work still relies on computationally expensive iterative pruning. In this work, we conduct an in-depth investigation of the structure of winning lottery tickets. Interestingly, we discover that there exist many lottery tickets that can achieve equally good accuracy much before the regular training schedule even finishes. We provide insights into the structure of these early winning tickets with supporting evidence. 1) Under stochastic gradient descent optimization, lottery ticket emerges when weight magnitude of a model saturates; 2) Pruning before the saturation of a model causes the loss of capability in learning complex patterns, resulting in the accuracy degradation. We employ the memorization capacity analysis to quantitatively confirm it, and further explain why gradual pruning can achieve better accuracy over the one-shot pruning. Based on these insights, we discover the early winning tickets for various ResNet architectures on both CIFAR10 and ImageNet, achieving state-of-the-art accuracy at a high pruning rate without expensive iterative pruning. In the case of ResNet50 on ImageNet, this comes to the winning ticket of 75.31% Top-1 accuracy at 80% pruning rate in only 22% of the total epochs for iterative pruning.
Overview:
The paper is dedicated to conducting an in-depth investigation of the structure of winning lottery tickets. The author provides supporting evidence for the structure of the early winning tickets: 1) lottery tickets emerge when the weight magnitude of a model saturates with SGD optimization. 2) pruning before model saturation may result in accuracy degradation. In the experiment part, they employ the memorization capacity analysis and discover the early wining tickets without expensive iterative pruning. The author also conducts extensive experiments with various ResNet architectures on both CIFAR 10 and ImageNet, achieving state-of-the-art results with only 1/5 of the total epochs for iterative pruning.
Strength Bullets:
1. The experiment organization is complete and convincing. Especially for the figure, it not only clearly shows that lottery tickets emerge much earlier before full training ends, but also shows the effect of rewinding. 
2. The author not reveal interesting observations, but also provides useful guidance. It is a complete logic chain that is also aligned with my intuition. For example, first, the author discusses the memorization capacity of different pruned models at different epochs. Then, they introduce a reasonable gradual pruning technique. Finally, they conduct experiments to confirm it.
3.  The early winning tickets in this paper achieve state-of-the-art results with only 1/5 of the total epochs for iterative pruning.
Weakness Bullets:
1. For lottery tickets, especially for early winning tickets, I think there is a lot of randomnesses. Thus, for the plot like figure 2, figure 5, they need to contain an error bar and the curve should be the average of tens of experiments. It will be more convincing if it decouples the randomness from the real patterns.
2. The description and organization of section 4 need to be more clear. For example, an algorithm pseudo code will definitely give readers a much more clear understanding of the early winning tickets finding strategy.
Question the authors don't answer which confuses me more:
Need more convincing analysis about the indicator - Hamming Distance 
Just as the comment I posted after Review 1, we would like to see more analysis about Hamming Distance between different winning tickets. The author mentioned in the following way:
''However, as we demonstrate in Fig. 3 of our paper, the mask-distance does not well characterize the winning tickets. E.g., the lottery tickets drawn at Epoch 120 and 200 have a mask distance of 0.082 in Fig. 3, which is much larger than the mask distance between Epoch 180 and 200. Whereas, all three tickets achieve comparably high accuracy as shown in Fig. 1, implying a shallow correlation between the accuracy and the mask distance''
As far as I know, this observation only tells us that the structure of the lottery tickets changes, which are drawn from 120 epochs to 180 epochs (although the maintain a similar accuracy). However, we can not conclude that the mask distance is not a reliable measure. As mention by the [EB] paper provided by the authors, you can use the rate of the distance change to indicate the early winning tickets. In this way, we can find winning tickets much earlier than authors' work.
To better address this question, I suppose the authors need to provide more analysis of the mask distance indicator. I think all three reviewers would like to see the results. A good indicator for early winning tickets is very important, otherwise authors' notion of early still involves quite a lot of training.
Recommendation:
Due to the unsolved important question, here is a weak reject.