Optimistic Adaptive Acceleration for Optimization
This paper considers a new variant of AMSGrad called Optimistic-AMSGrad. AMSGrad is a popular adaptive gradient based optimization algorithm that is widely used in training deep neural networks. The new variant assumes that mini-batch gradients in consecutive iterations have some underlying structure, which makes the gradients sequentially predictable. By exploiting the predictability and some ideas from Optimistic Online learning, the proposed algorithm can accelerate the convergence and also enjoys a tighter regret bound. We evaluate Optimistic-AMSGrad and AMSGrad in terms of various performance measures (i.e., training loss, testing loss, and classification accuracy on training/testing data), which demonstrate that Optimistic-AMSGrad improves AMSGrad.
This paper studies an optimistic variant of AMSGrad algorithm, where an estimate of the future gradient is incorporated into the optimization problem. The main claim is that when we have good enough (distance from the ground truth is small) estimate of the unknown gradient, the proposed algorithm will enjoy lower regret. Theoretical results are provided and experiments are conducted to compare the proposed algorithm with baselines. The idea seems to be not very novel since the optimistic optimization techniques are borrowed directly from the online optimization field, while it is still interesting to see this kind of work and to see its comparison with existing algorithms in experiments. However, the comparison seems to be not fair both in theory and experiments. In the second paragraph of Section 2.1, you use to denote an estimate of the loss function. But later (In the third equation) you use to denote the guess of gradient vector. The notation is reloaded without any description, which makes the presentation confusing. In addition, at the end of this paragraph, you mentioned that even when is far away from , the regret of an optimistic algorithm is just a constant factor of non-optimistic one. This seems not rigorous since it is true only when the divergence of from is in constant order. Can you explain why in line 8 of Algorithm 2, you use ½ to update instead of ? A discussion about these choices should be added to the description of algorithm. In the first equation on page 5, the second term on the R.H.S. of the first equation misses a factor of . Moreover, the second equation should be inequality. In the comparison of equation (2) and (3), I think it should be pointed out that when the gradient has a sparse structure, the regret of the optimistic-AMSGrad in (2) seems to be worse than that of the original AMSGrad. The optimistic algorithm seems to cost more computation in order to estimate the unknown gradient in advance. In the experiment part, you used the last few iterations to estimate the guess of gradient in the next step. But it seems that the comparison with is not consistent in many plots since I expected a larger will lead to more accurate estimate. The experiment results seem to be not convincing. In particular, in Fig. 1,2 and others, the training loss of AMSGrad is far away from zero, which implies that the algorithm is not fully optimized. Therefore, it is hard to draw any meaningful conclusion from the current experiments. In footnote 1, “had been known” -> “had known” ====after rebuttal The authors did not provide satisfying response neither submit any revision to address the questions. I will keep my rating.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Optimistic Adaptive Acceleration for Optimization
This paper considers a new variant of AMSGrad called Optimistic-AMSGrad. AMSGrad is a popular adaptive gradient based optimization algorithm that is widely used in training deep neural networks. The new variant assumes that mini-batch gradients in consecutive iterations have some underlying structure, which makes the gradients sequentially predictable. By exploiting the predictability and some ideas from Optimistic Online learning, the proposed algorithm can accelerate the convergence and also enjoys a tighter regret bound. We evaluate Optimistic-AMSGrad and AMSGrad in terms of various performance measures (i.e., training loss, testing loss, and classification accuracy on training/testing data), which demonstrate that Optimistic-AMSGrad improves AMSGrad.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper studies an optimistic variant of AMSGrad algorithm, where an estimate of the future gradient is incorporated into the optimization problem. The main claim is that when we have good enough (distance from the ground truth is small) estimate of the unknown gradient, the proposed algorithm will enjoy lower regret. Theoretical results are provided and experiments are conducted to compare the proposed algorithm with baselines. The idea seems to be not very novel since the optimistic optimization techniques are borrowed directly from the online optimization field, while it is still interesting to see this kind of work and to see its comparison with existing algorithms in experiments. However, the comparison seems to be not fair both in theory and experiments. In the second paragraph of Section 2.1, you use to denote an estimate of the loss function. But later (In the third equation) you use to denote the guess of gradient vector. The notation is reloaded without any description, which makes the presentation confusing. In addition, at the end of this paragraph, you mentioned that even when is far away from , the regret of an optimistic algorithm is just a constant factor of non-optimistic one. This seems not rigorous since it is true only when the divergence of from is in constant order. Can you explain why in line 8 of Algorithm 2, you use ½ to update instead of ? A discussion about these choices should be added to the description of algorithm. In the first equation on page 5, the second term on the R.H.S. of the first equation misses a factor of . Moreover, the second equation should be inequality. In the comparison of equation (2) and (3), I think it should be pointed out that when the gradient has a sparse structure, the regret of the optimistic-AMSGrad in (2) seems to be worse than that of the original AMSGrad. The optimistic algorithm seems to cost more computation in order to estimate the unknown gradient in advance. In the experiment part, you used the last few iterations to estimate the guess of gradient in the next step. But it seems that the comparison with is not consistent in many plots since I expected a larger will lead to more accurate estimate. The experiment results seem to be not convincing. In particular, in Fig. 1,2 and others, the training loss of AMSGrad is far away from zero, which implies that the algorithm is not fully optimized. Therefore, it is hard to draw any meaningful conclusion from the current experiments. In footnote 1, “had been known” -> “had known” ====after rebuttal The authors did not provide satisfying response neither submit any revision to address the questions. I will keep my rating.
assistant
This paper studies an optimistic variant of AMSGrad algorithm, where an estimate of the future gradient is incorporated into the optimization problem. The main claim is that when we have good enough (distance from the ground truth is small) estimate of the unknown gradient, the proposed algorithm will enjoy lower regret. Theoretical results are provided and experiments are conducted to compare the proposed algorithm with baselines. The idea seems to be not very novel since the optimistic optimization techniques are borrowed directly from the online optimization field, while it is still interesting to see this kind of work and to see its comparison with existing algorithms in experiments. However, the comparison seems to be not fair both in theory and experiments. In the second paragraph of Section 2.1, you use to denote an estimate of the loss function. But later (In the third equation) you use to denote the guess of gradient vector. The notation is reloaded without any description, which makes the presentation confusing. In addition, at the end of this paragraph, you mentioned that even when is far away from , the regret of an optimistic algorithm is just a constant factor of non-optimistic one. This seems not rigorous since it is true only when the divergence of from is in constant order. Can you explain why in line 8 of Algorithm 2, you use ½ to update instead of ? A discussion about these choices should be added to the description of algorithm. In the first equation on page 5, the second term on the R.H.S. of the first equation misses a factor of . Moreover, the second equation should be inequality. In the comparison of equation (2) and (3), I think it should be pointed out that when the gradient has a sparse structure, the regret of the optimistic-AMSGrad in (2) seems to be worse than that of the original AMSGrad. The optimistic algorithm seems to cost more computation in order to estimate the unknown gradient in advance. In the experiment part, you used the last few iterations to estimate the guess of gradient in the next step. But it seems that the comparison with is not consistent in many plots since I expected a larger will lead to more accurate estimate. The experiment results seem to be not convincing. In particular, in Fig. 1,2 and others, the training loss of AMSGrad is far away from zero, which implies that the algorithm is not fully optimized. Therefore, it is hard to draw any meaningful conclusion from the current experiments. In footnote 1, “had been known” -> “had known” ====after rebuttal The authors did not provide satisfying response neither submit any revision to address the questions. I will keep my rating.