ABSTRACT (Frankle & Carbin, 2019) shows that there exist winning tickets (small but critical subnetworks) for dense, randomly initialized networks, that can be trained alone to achieve comparable accuracies to the latter in a similar number of iterations. However, the identification of these winning tickets still requires the costly trainprune-retrain process, limiting their practical benefits. In this paper, we discover for the first time that the winning tickets can be identified at the very early training stage, which we term as early-bird (EB) tickets, via low-cost training schemes (e.g., early stopping, low-precision training) at large learning rates. Our finding of EB tickets is consistent with recently reported observations that the key connectivity patterns of neural networks emerge early. Furthermore, we propose a mask distance metric that can be used to identify EB tickets with low computational overhead, without needing to know the true winning tickets that emerge after the full training. Finally, we leverage the existence of EB tickets and the proposed mask distance to develop efficient training methods, which are achieved by first identifying EB tickets via low-cost schemes, and then continuing to train merely the EB tickets towards the target accuracy. Experiments based on various deep networks and datasets validate: 1) the existence of EB tickets, and the effectiveness of mask distance in efficiently identifying them; and 2) that the proposed efficient training via EB tickets can achieve up to 4.7× energy savings while maintaining comparable or even better accuracy, demonstrating a promising and easily adopted method for tackling cost-prohibitive deep network training.
This paper presents a method to speed up training of deep neural networks. The main contribution is a method to quickly identify winning lottery tickets (denoted early-bird, or EB by the authors), without running the model to convergence. The authors present interesting preliminary experiments that motivate their method, and show that it works on two image recognition datasets using two models.
This paper addresses an under-explored, but very important problem in AI: the increasing cost of training models. The authors present interesting evidence about the potential to detect EBs early on. The experiments presented in Figures 1 and 3 are convincing and will be of interest to the community. The proposed method seems to work, at least on the setups explored by the authors. I am leaning towards acceptance, but am concerned with the following: 
1. The authors experiment with a limited set of datasets (CIFAR-10 is a relatively easy task), and with a set of non-competitive baselines (SOTA for CIFAR-10/100 is 99%/91.3%, see https://benchmarks.ai/cifar-10{,0}). I would have liked to see whether the proposed method translates to harder datasets and stronger models.
2. I might be missing something here, but to the best of my understanding the large learning rate part (page 4) does not demonstrate the benefits of increasing the learning rate, but the problems with *decreasing* it. The two might seem like the same thing, but in fact they're not: the authors claim the [80,120] policy is standard, and use it when training the subnetwork, so showing that [0,100] is inferior does not present a way to improve over the current approach, but evidence that the other approaches are inferior.
Other questions: 
1. In Figure 1, it seems that the extracted subnetworks are doing very well even after 0 epochs. Does this mean that a trained version of a random subnetwork could reach within 1-2 points of the unpruned model? or is it pruned after training for 1 epoch?
2. If I understand correctly, Figure 5 should be illustrating the proposed method, which automatically identifies the early stopping point. In that case, I am not sure why the plot is a function of the epoch.
3. Do the authors have any intuition as to the sharp decrease in the 70% graph in Figures 1 and 2 around epoch 50?
Writing: 
1. The language used by the authors is sometimes exaggerated. Expressions such as "bold guess" (section 3.2), "innovative ... scheme" (section 4) and comparisons to Winston Churchill would be better left out of the manuscript. 
2. Typos and such: 
-- several across the paper. For instance: 
- Intro: After *bring* identified (should be "being")
- Related work: when training *it* isolation (in)
-- Missing venue for Frankle and Corbin (2019)