We study the problem of fitting task-specific learning rate schedules from the perspective of hyperparameter optimization. This allows us to explicitly search for schedules that achieve good generalization. We describe the structure of the gradient of a validation error w.r.t. the learning rate, the hypergradient, and based on this we introduce a novel online algorithm. Our method adaptively interpolates between the recently proposed techniques of Franceschi et al. (2017) and Baydin et al. (2018), featuring increased stability and faster convergence. We show empirically that the proposed method compares favorably with baselines and related methods in terms of final test accuracy.
In this paper, the authors introduce a hypergradient optimization algorithm for finding learning rate schedules that maximize test set accuracy. The proposed algorithm adaptively interpolates between two recently proposed hyperparameter optimization algorithms and performs comparably in terms of convergence and generalization with these baselines.
Overall the paper is interesting, although I found it a bit dense and hard to read. I frequently found myself having to scroll to different parts of the paper to remind myself of the notation used and the definition of the different matrices. This makes it harder to evaluate the paper properly. The proposed algorithm seems interesting however, and the experimental results look quite impressive.
I have a few concerns regarding the experiments however, which explains my score:
1. In figure 2, does MARTHE diverge for values of beta greater than 1e-4? This seems to indicate that MARTHE is somehow more sensitive to beta than the other variations used. Do the authors have any intuition about what might be causing this behavior?
2. The initial learning rate for SGDM and Adam was fixed at certain values for all experiments. Why is this a reasonable thing to do? It feels like MARTHE should be compared to SGDM and Adam at least when the initial learning rate is tuned for these properly. Otherwise, it doesn't feel like a fair evaluation? To the best of my knowledge, the final achieved accuracies achieved with MARTHE however seem quite competitive with the best results typically reached with tuned SGDM on the convolutional nets used in the paper.
3. The learning rate schedules found by MARTHE seem to be somewhat counterintuitive. While an initial increase matches the heuristic of warmup learning rates frequently used when training convnets, the algorithms seems to decrease down the learning rate after that even quicker than what the greedy algorithm HD does. Do the authors have any intuition why this can lead to such a big improvement in performance over HD?
4. Is it possible to provide some sort of estimate of how much computation MARTHE requires compared to a single SGDM run? How feasible is to test this algorithm on a bigger classification model on ImageNet?
I think this paper is borderline, although I am leaning towards accepting it given the impressive empirical results. It would really improve the paper if the readability was improved, as well as if larger experimental results were included.
====================================
Edit after rebuttal:
I thank the authors for their response. I am happy with their response and am sticking to my score.