Understanding Short-Horizon Bias in Stochastic Meta-Optimization
Careful tuning of the learning rate, or even schedules thereof, can be crucial to effective neural net training. There has been much recent interest in gradient-based meta-optimization, where one tunes hyperparameters, or even learns an optimizer, in order to minimize the expected loss when the training procedure is unrolled. But because the training procedure must be unrolled thousands of times, the meta-objective must be defined with an orders-of-magnitude shorter time horizon than is typical for neural net training. We show that such short-horizon meta-objectives cause a serious bias towards small step sizes, an effect we term short-horizon bias. We introduce a toy problem, a noisy quadratic cost function, on which we analyze short-horizon bias by deriving and comparing the optimal schedules for short and long time horizons. We then run meta-optimization experiments (both offline and online) on standard benchmark datasets, showing that meta-optimization chooses too small a learning rate by multiple orders of magnitude, even when run with a moderately long time horizon (100 steps) typical of work in the area. We believe short-horizon bias is a fundamental problem that needs to be addressed if meta-optimization is to scale to practical neural net training regimes.
 The paper discusses the problems of meta optimization with small look-ahead: do small runs bias the results of tuning? The result is yes and the authors show how differently the tuning can be compared to tuning the full run. The Greedy schedules are far inferior to hand-tuned schedules as they focus on optimizing the large eigenvalues while the small eigenvalues can not be "seen" with a small lookahead. The authors show that this effect is caused by the noise in the obective function. pro: - Thorough discussion of the issue with theoretical understanding on small benchmark functions as well as theoretical work - Easy to read and follow cons: -Small issues in presentation: * Figure 2 "optimal learning rate" -> "optimal greedy learning rate", also reference to Theorem 2 for increased clarity. * The optimized learning rate in 2.3 is not described. This reduces reproducibility. * Figure 4 misses the red trajectories, also it would be easier to have colors on the same (log?)-scale. The text unfortunately does not explain why the loss function looks so vastly different with different look-ahead. I would assume from the description that the colors are based on the final loss values obtaine dby choosing a fixed pair of decay exponent and effective LR. Typos and notation: page 7 last paragraph: "We train the all" -> We train all notation page 5: i find .nabla_{.theta_i} confusing when .theta_i is a scalar, i would propose .frac{.partial}{.partial .theta_i} page 2: "But this would come at the expense of long-term optimization process": at this point of the paper it is not clear how or why this should happen. Maybe add a sentence regarding the large/Small eigenvalues?
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Understanding Short-Horizon Bias in Stochastic Meta-Optimization
Careful tuning of the learning rate, or even schedules thereof, can be crucial to effective neural net training. There has been much recent interest in gradient-based meta-optimization, where one tunes hyperparameters, or even learns an optimizer, in order to minimize the expected loss when the training procedure is unrolled. But because the training procedure must be unrolled thousands of times, the meta-objective must be defined with an orders-of-magnitude shorter time horizon than is typical for neural net training. We show that such short-horizon meta-objectives cause a serious bias towards small step sizes, an effect we term short-horizon bias. We introduce a toy problem, a noisy quadratic cost function, on which we analyze short-horizon bias by deriving and comparing the optimal schedules for short and long time horizons. We then run meta-optimization experiments (both offline and online) on standard benchmark datasets, showing that meta-optimization chooses too small a learning rate by multiple orders of magnitude, even when run with a moderately long time horizon (100 steps) typical of work in the area. We believe short-horizon bias is a fundamental problem that needs to be addressed if meta-optimization is to scale to practical neural net training regimes.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The paper discusses the problems of meta optimization with small look-ahead: do small runs bias the results of tuning? The result is yes and the authors show how differently the tuning can be compared to tuning the full run. The Greedy schedules are far inferior to hand-tuned schedules as they focus on optimizing the large eigenvalues while the small eigenvalues can not be "seen" with a small lookahead. The authors show that this effect is caused by the noise in the obective function. pro: - Thorough discussion of the issue with theoretical understanding on small benchmark functions as well as theoretical work - Easy to read and follow cons: -Small issues in presentation: * Figure 2 "optimal learning rate" -> "optimal greedy learning rate", also reference to Theorem 2 for increased clarity. * The optimized learning rate in 2.3 is not described. This reduces reproducibility. * Figure 4 misses the red trajectories, also it would be easier to have colors on the same (log?)-scale. The text unfortunately does not explain why the loss function looks so vastly different with different look-ahead. I would assume from the description that the colors are based on the final loss values obtaine dby choosing a fixed pair of decay exponent and effective LR. Typos and notation: page 7 last paragraph: "We train the all" -> We train all notation page 5: i find .nabla_{.theta_i} confusing when .theta_i is a scalar, i would propose .frac{.partial}{.partial .theta_i} page 2: "But this would come at the expense of long-term optimization process": at this point of the paper it is not clear how or why this should happen. Maybe add a sentence regarding the large/Small eigenvalues?
assistant
 The paper discusses the problems of meta optimization with small look-ahead: do small runs bias the results of tuning? The result is yes and the authors show how differently the tuning can be compared to tuning the full run. The Greedy schedules are far inferior to hand-tuned schedules as they focus on optimizing the large eigenvalues while the small eigenvalues can not be "seen" with a small lookahead. The authors show that this effect is caused by the noise in the obective function. pro: - Thorough discussion of the issue with theoretical understanding on small benchmark functions as well as theoretical work - Easy to read and follow cons: -Small issues in presentation: * Figure 2 "optimal learning rate" -> "optimal greedy learning rate", also reference to Theorem 2 for increased clarity. * The optimized learning rate in 2.3 is not described. This reduces reproducibility. * Figure 4 misses the red trajectories, also it would be easier to have colors on the same (log?)-scale. The text unfortunately does not explain why the loss function looks so vastly different with different look-ahead. I would assume from the description that the colors are based on the final loss values obtaine dby choosing a fixed pair of decay exponent and effective LR. Typos and notation: page 7 last paragraph: "We train the all" -> We train all notation page 5: i find .nabla_{.theta_i} confusing when .theta_i is a scalar, i would propose .frac{.partial}{.partial .theta_i} page 2: "But this would come at the expense of long-term optimization process": at this point of the paper it is not clear how or why this should happen. Maybe add a sentence regarding the large/Small eigenvalues?