In most practical settings and theoretical analyses, one assumes that a model can be trained until convergence. However, the growing complexity of machine learning datasets and models may violate such assumptions. Indeed, current approaches for hyper-parameter tuning and neural architecture search tend to be limited by practical resource constraints. Therefore, we introduce a formal setting for studying training under the non-asymptotic, resource-constrained regime, i.e., budgeted training. We analyze the following problem: "given a dataset, algorithm, and fixed resource budget, what is the best achievable performance?" We focus on the number of optimization iterations as the representative resource. Under such a setting, we show that it is critical to adjust the learning rate schedule according to the given budget. Among budget-aware learning schedules, we find simple linear decay to be both robust and high-performing. We support our claim through extensive experiments with state-of-the-art models on ImageNet (image classification), Kinetics (video classification), MS COCO (object detection and instance segmentation), and Cityscapes (semantic segmentation). We also analyze our results and find that the key to a good schedule is budgeted convergence, a phenomenon whereby the gradient vanishes at the end of each allowed budget. We also revisit existing approaches for fast convergence and show that budgetaware learning schedules readily outperform such approaches under (the practical but under-explored) budgeted training setting.
Pros:
The paper is clearly written. It provides an interesting perspective for training neural networks under resource constraints. The problem setting is novel. The proposed solution is simply decaying learning rate linearly from the initial value to zero during training, which is parameter free. 
Cons:
- As the authors are advocating using linear scaling schedule, I would like to see whether it has some clear advantages over other schedules, but it is not quite clear. For example, we can still see step based schedule has better performance in 2 of the 4 tasks in Table 2. Poly and Cosine schedule is also better in some of the budgets in Figure 2. 
- The comparison in Figure 2 and Table 2 is not very convincing without considering the variance of different trails. It is not clear whether the advantage is caused by learning rate schedule or randomness. It is better to report the mean and variance for multiple trials. Ideally, it would be better to performance significance test.
- I would like to see other lr schedules in Table 2. As shown in Figure 2, step based schedule is not the top3 schedules for CIFAR-10.
- As shown in Table 3, the proposed method has to wait until the end of training to get the best performing model, while step based schedule can find the best model around 90% training. The author argues that the proposed method does not need to perform validation test for each checkpoint and reduce the computation cost,  however, on the other side, this means early stopping is not able to use for linear scaling based schedule, which could be very useful when the training budget is large enough and evaluation is cheap.
- My major concern for this work is a lack of deeper understanding about the reason why linear LR schedule works better, if any. It would be stronger with such understandings. The authors try to provide an explanation from the relationship between learning rate and gradient magnitudes, but no clear conclusion is given. As noted in [1,2,3], when weight decay is used in training and BN layers are used, the weight magnitude is also decreasing, so is the gradient norms. But the weight norms or gradient norms does not mean too much due to scale invariance. I would like to see when no weight decay is used and whether there is any correlation between the learning rate and gradient norms. 
- What is the lr decay unit for linear schedule? Is it decaying per epoch or per mini-batch? If epoch based lr decay is used, it is essentially step-based lr decay with many steps. Then when the number of epochs is three, the step decay method (lr decays at epoch 1 and 2) and the linear decay method are actually almost equivalent. 
- I would expect the convergence to be related with number of iterations. When the number of iterations is not long enough, neither linear lr decay or step based decay will work. It would be better if the author can investigate when linear schedule starts to outperforms step based decay in terms of epochs or iterations. Since different batch size results in different number of iterations, I would expect the difference between two schedules for small batch size at the early stage of training would be less in comparison with large batch training, especially when the number of iterations is enough.
- Different initial learning rate may also results in different behaviour. We often see some learning curves with larger initial learning rate converges faster at the beginning but yields similar generalization error at the end of training. On the other hand, the author only compared different schedules with single initial learning rate. Image when the initial learning rate is small, there would not be too much difference for different schedules. Actually the linear decay schedule changes may simply find a good learning rate during training as long as the initial learning rate is larger than the optimal one.
[1] Dinh et al, Sharp minima can generalize for deep nets, ICML 2017
[2] Li et al, Visualizing the Loss Landscape of Neural Nets, NIPS 2018
[3] van Laarhoven, L2 regularization versus batch and weight normalization, NIPS 2017
----- update after rebuttal ------
The authors' rebuttal addressed some of my concerns.  I think early stopping is still an important feature to save compute, especially for HPO, which could limit the usage of the proposed method (cannot stop earlier). If the authors believe practitioners are used to train model in full budget without early stopping to guarantee the best performance, then why would people try budged training with worse performance? I hope the authors could make the limitation clear. My major concern about the reason for why linear scaling schedule is better is still not clear, which makes the contribution kind of weak. Nevertheless,  the problem setting and the observations could be beneficial to the community for further discussion, So I raised my score to weak accept.
minor: I see exactly the same variance values for step and linear methods int the top rows of Table 12, is this a mistake?