UNDERSTANDING SHORT-HORIZON BIAS IN STOCHASTIC META-OPTIMIZATION
Careful tuning of the learning rate, or even schedules thereof, can be crucial to effective neural net training. There has been much recent interest in gradient-based meta-optimization, where one tunes hyperparameters, or even learns an optimizer, in order to minimize the expected loss when the training procedure is unrolled. But because the training procedure must be unrolled thousands of times, the metaobjective must be defined with an orders-of-magnitude shorter time horizon than is typical for neural net training. We show that such short-horizon meta-objectives cause a serious bias towards small step sizes, an effect we term short-horizon bias. We introduce a toy problem, a noisy quadratic cost function, on which we analyze short-horizon bias by deriving and comparing the optimal schedules for short and long time horizons. We then run meta-optimization experiments (both offline and online) on standard benchmark datasets, showing that meta-optimization chooses too small a learning rate by multiple orders of magnitude, even when run with a moderately long time horizon (100 steps) typical of work in the area. We believe short-horizon bias is a fundamental problem that needs to be addressed if metaoptimization is to scale to practical neural net training regimes.
This paper studies the issue of truncated backpropagation for meta-optimization. Backpropagation through an optimization process requires unrolling the optimization, which due to computational and memory constraints, is typically restricted or truncated to a smaller number of unrolled steps than we would like.
This paper highlights this problem as a fundamental issue limiting meta-optimization approaches. The authors perform a number of experiments on a toy problem (stochastic quadratics) which is amenable to some theoretical analysis as well as a small fully connected network trained on MNIST.  
(side note: I was assigned this paper quite late in the review process, and have not carefully gone through the derivations--specifically Theorems 1 and 2).
The paper is generally clear and well written.
Major comments
-------------------------
I was a bit confused why 1000 SGD+mom steps pre-training steps were needed. As far as I can tell, pre-training is not typically done in the other meta-optimization literature? The authors suggest this is needed because "the dynamics of training are different at the very start compared to later stages", which is a bit vague. Perhaps the authors can expand upon  this point?
The conclusion suggests that the difference in greedy vs. fully optimized schedule is due to the curvature (poor scaling) of the objective--but Fig 2. and earlier discussion talked about the noise in the objective as introducing the bias (e.g. from earlier in the paper, "The noise in the problem adds uncertainty to the objective, resulting in failures of greedy schedule"). Which is the real issue, noise or curvature? Would running the problem on quadratics with different condition numbers be insightful?
Minor comments
-------------------------
The stochastic gradient equation in Sec 2.2.2 is missing a subscript: "h_i" instead of "h"
It would be nice to include the loss curve for a fixed learning rate and momentum for the noisy quadratic in Figure 2, just to get a sense of how that compares with the greedy and optimized curves.
It looks like there was an upper bound constraint placed on the optimized learning rate in Figure 2--is that correct? I couldn't find a mention of the constraint in the paper. (the optimized learning rate remains at 0.2 for the first ~60 steps)?
Figure 2 (and elsewhere): I would change 'optimal' to 'optimized' to distinguish it from an optimal curve that might result from an analytic derivation. 'Optimized' makes it more clear that the curve was obtained using an optimization process.
Figure 2: can you change the line style or thickness so that we can see both the red and blue curves for the deterministic case? I assume the red curve is hiding beneath the blue one--but it would be good to see this explicitly.
Figure 4 is fantastic--it succinctly and clearly demonstrates the problem of truncated unrolls. I would add a note in the caption to make it clear that the SMD trajectories are the red curves, e.g.: "SMD trajectories (red) during meta-optimization of initial effective ...". I would also change the caption to use "meta-training losses" instead of "training losses" (I believe those numbers are for the meta-loss, correct?). Finally, I would add a colorbar to indicate numerical values for the different grayscale values.
Some recent references that warrant a mention in the text:
- both of these learn optimizers using longer numbers of unrolled steps:
Learning gradient descent: better generalization and longer horizons, Lv et al, ICML 2017
Learned optimizers that scale and generalize, Wichrowska et al, ICML 2017
- another application of unrolled optimization:
Unrolled generative adversarial networks, Metz et al, ICLR 2017
In the text discussing Figure 4 (middle of pg. 8) , "which is obtained by using..." should be "which are obtained by using..."
In the conclusion, "optimal for deterministic objective" should be "deterministic objectives"