Training neural networks on image datasets generally require extensive experimentation to find the optimal learning rate regime. Especially, for the cases of adversarial training or for training a newly synthesized model, one would not know the best learning rate regime beforehand. We propose an automated algorithm for determining the learning rate trajectory, that works across datasets and models for both natural and adversarial training, without requiring any dataset/model specific tuning. It is a stand-alone, parameterless, adaptive approach with no computational overhead. We theoretically discuss the algorithm's convergence behavior. We empirically validate our algorithm extensively. Our results show that our proposed approach consistently achieves top-level accuracy compared to SOTA baselines in the literature in natural as well as adversarial training.
The paper considers the problem of automated adaptation of learning rate during (deep) neural network training. The use cases described are standard and adversarial training for image classification. Given the wide use of DNNs in computer vision (and other areas), learning rate tuning is clearly an important problem and is being actively researched.
The proposed learning rate adaptation procedure consists of a straightforward combination of learning rate halving/doubling and model checkpointing. Experimental results from implementing the adaptive learning rule for standard and adversarial training on CIFAR are provided. Multiple architectures are tested in each setting. The paper claims a primary advantage of the proposed learning rule to be that it requires no tuning as opposed to other rules such as SGD, Adam.
My decision is to reject the paper due to methodological issues with the experiments and lack of evidence wrt/ dataset variety. The paper should be considered a work-in-progress that may have potential in a more focused setting, e.g., adversarial training as described in the paper.
***
The major claim of the proposed algorithm not requiring any manual tuning is technically true but misleading. The algorithm does have parameters (SGD momentum, batch size, initial learning rate, patience) with values that were set somehow. In fact, a major methodological issue with the experiments is that the reader does not know if the datasets were used to both set these values and to assess performance, i.e., there are no obvious "held-out" datasets. Also, there is no rigorous or even informal justification of the settings. It could be that the paper is arguing that the specific values will result in competitive, if not better, performance than baselines across a variety of datasets - unfortunately, only two datasets are utilized in the experiments, and one, CIFAR10, is not considered challenging. This leads to the second issue with the paper: the experimental validation is not extensive wrt/ datasets which is significant given that the form of the evidence for the proposed method is almost entirely empirical.
Additionally, I don't agree that competitor algorithms should not be tuned b/c the proposed method does not require tuning. Even if the proposed method does not require tuning (as stated previously, I don't believe this to be accurate), that does not imply a fair comparison precludes tuning competitors via, e.g., cross validation. The only relevant quantities are final test-set performance and total training time/resources required. 
The well-known interdependence between learning rate and batchsize as noted in e.g., Hoffer et al. (2018), is not addressed by the experiments. Batchsizes in the experiments vary, but no justification is provided for how these are selected.
Finally, the paper is unfinished as some experimental runs were not complete at the time of submission.
On the positive side, the general point about the necessity of learning rate tuning for adversarial training (described in the fourth paragraph of the introduction) is a very good one, and there may be an opportunity for a more focused application of the proposed algorithm perhaps among further datasets and considering additional, alternative attacks.
***
Suggestions for improvement / questions (related to decision):
* It should not be a challenge to find more image classification datasets to include in the experimental comparison: SVHN, Fashion MNIST, Imagenet, ... Using these, the paper can either follow the standard train/test methodology *across datasets*, i.e., split the meta-dataset into train/test, and/or provide a more compelling body of evidence for the proposed method. Also, the performance dependence on batchsize amongst the proposed algorithm and competitors should be investigated experimentally.
* The Baydin et al. (2018) algorithm should be added to the set of competitors since it would provide a relatively easy* reference point wrt/ "hypergradient" approaches. I don't agree with the statement in the related work section that this entails "additional computation of gradients." *In the sense that the rule should be straightforward to implement.
* The convergence analysis assumption that the optimal oracle SGD follows typical learning rate regimes motivated by loss plateauing seems to be in direct contradiction to the sentiment expressed in the cited Hoffer et al. (2018) paper that such "rules of thumb" may be misguided. Can the authors discuss the appropriateness of their assumption wrt/ this point? Also, in the convergence analysis, the phrase "in expectation" is used twice. This has a specific probabilistic meaning, but appears to be used heuristically in this section. Can the authors clarify whether this usage is informal or formal? If the latter is true, it would be better to provide a more formal convergence argument that explicitly takes the inherent randomness into account.
***
Editorial comments (not related to decision):
* Introduction: The first two sentences of the second paragraph, particularly the second, would do well to have an accompanying reference or references.
* Proposed method: Even as an informal statement, the second sentence of the second paragraph under the Phase 2 sub-heading is problematic. The proposed method does not "resist" lowering the learning rate "for as long as possible" so much as it doesn't lower the learning rate for a fixed number of epochs (algorithm parameter).
* "Adversarial training" section (5.4): The paper assumes the reader is familiar with the terms "FGSM", "white box", and parameters \epsilon and \alpha since these are referred to w/o description. Perhaps a short (2-3 sentence) description of the adversarial scenario could be added?
* Experiments: It would be good for the paper to include RMSProp and Adagrad results to the experimental tables as these rules are both readily available for use and widely used.
* Experiments: Is the reporting of the peak accuracy standard in the literature?
* Experiments: I want to give the paper credit for performance on CIFAR100, but this is difficult without explicit points of comparison. This can be easily remedied by including SOTA performance values (along with appropriate references) in the tables or text.
* (Potential) Typos:
	Proposed method algorithm description:
		Requirements has a weight decay parameter which seems strange given that the algorithm is performing automated learning rate adaptation...
		The epoch counter is incremented in line 5, but not reset prior to Phase 2. Does this mean that Phase 1 training epochs are counted toward the total (T)?
		Line 7 should be \eta_t <- \eta_0 / 2.
		The patience counter in line 15 is not utilized below.
		Line 23 could/should be an else statement.