Amortized Nesterov*s Momentum: Robust and Lightweight Momentum for Deep Learning
Stochastic Gradient Descent (SGD) with Nesterov*s momentum is a widely used optimizer in deep learning, which is observed to have excellent generalization performance. However, due to the large stochasticity, SGD with Nesterov*s momentum is not robust, i.e., its performance may deviate significantly from the expectation. In this work, we propose Amortized Nesterov*s Momentum, a special variant of Nesterov*s momentum which has more robust iterates, faster convergence in the early stage and higher efficiency. Our experimental results show that this new momentum achieves similar (sometimes better) generalization performance with little-to-no tuning. In the convex case, we provide optimal convergence rates for our new methods and discuss how the theorems explain the empirical results.
The authors proposed Amortized Nesterov’s Momentum, a variant of Nesterov’s momentum that utilizes several past iterates, instead of one iterate, to provide the momentum. The goal is to have more robust iterates, faster convergence in the early stage and higher efficiency. The authors designed two different realizations, AM1-SGD and AM2-SGD. Comments: My major concern for this paper is its unconvincing motivation and experiment results, especially when the approach is designed for deep learning. The motivation for the proposed approach is not quite convincing. The authors said that “due to the large stochasticity, SGD with Nesterov’s momentum is not robust...This increased uncertainty slows down its convergence especially in the early stage of training and makes its single run less trustworthy” For image classification, Nesterov momentum is very popular and the final convergence values of different trails seems to be similar. It would be more convincing if the authors can provide practical evidence for supporting this claim. It was claimed that Amortized Nesterov’s Momentum has “higher efficiency and faster convergence in the early stage without losing the good generalization performance”. What is the benefit or advantage for having faster early convergence without improving the final generalization performance? The authors claim that “M-SGD2 is more robust and achieves slightly better performance”, in Figure 1a, however, it is really hard to tell the difference between M-SGD2 and M-SGD from Figure 1a. The efficiency improvement in page 4 is really hard to follow for comparison with Algorithm 1 in page 3. Though m > 2 could reduce the number of operations in step 5 and 6, I don’t think this is a computational bottleneck. I believe these updates should be very fast in comparison with forward and gradient calculation. Making the 1% computation 50% faster does not mean more efficient. I would like to know how much computation cost can be saved with this modification. On the other hand, adding one more auxiliary buffer (scaled momentum) could significantly impact the training as the memory is often the limit. In section 3.1, what is “identical iteration”? It is hard to compare AM2-SGD with AM1-SGD. It would be easier to follow if the side-by-side algorithm comparison can be shown early. The section 4’s theoretical analysis based on the convex composite problem is not quite convincing. I am not sure how these results are related with the deep learning applications. In the experiment section, the comparison of AM1/2-SGD with other baselines seems not quite consistent. The authors first state that they use all 0.1 learning rate and 0.9 momentum for all methods, however, the setting for M-SGD is using 0.99 momentum and different learning rate schedule. This makes the comparison not very meaningful, while AM1-SGD and AM2-SGD do not use learning rate restart. With so many differences, the advantage of AM1-SGD and AM2-SGD are not that different with M-SGD. In the task of ImageNet-152, M-SGD even is better than AM1-SGD. This makes the conclusion that “AM1-SGD has a lightweight design, which can serve as an excellent replacement for M-SGD in large-scale deep learning tasks” not quite valid. Minor: The author may assume readers maybe familiar Katyusha momentum, I think there may need more background about it.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Amortized Nesterov*s Momentum: Robust and Lightweight Momentum for Deep Learning
Stochastic Gradient Descent (SGD) with Nesterov*s momentum is a widely used optimizer in deep learning, which is observed to have excellent generalization performance. However, due to the large stochasticity, SGD with Nesterov*s momentum is not robust, i.e., its performance may deviate significantly from the expectation. In this work, we propose Amortized Nesterov*s Momentum, a special variant of Nesterov*s momentum which has more robust iterates, faster convergence in the early stage and higher efficiency. Our experimental results show that this new momentum achieves similar (sometimes better) generalization performance with little-to-no tuning. In the convex case, we provide optimal convergence rates for our new methods and discuss how the theorems explain the empirical results.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The authors proposed Amortized Nesterov’s Momentum, a variant of Nesterov’s momentum that utilizes several past iterates, instead of one iterate, to provide the momentum. The goal is to have more robust iterates, faster convergence in the early stage and higher efficiency. The authors designed two different realizations, AM1-SGD and AM2-SGD. Comments: My major concern for this paper is its unconvincing motivation and experiment results, especially when the approach is designed for deep learning. The motivation for the proposed approach is not quite convincing. The authors said that “due to the large stochasticity, SGD with Nesterov’s momentum is not robust...This increased uncertainty slows down its convergence especially in the early stage of training and makes its single run less trustworthy” For image classification, Nesterov momentum is very popular and the final convergence values of different trails seems to be similar. It would be more convincing if the authors can provide practical evidence for supporting this claim. It was claimed that Amortized Nesterov’s Momentum has “higher efficiency and faster convergence in the early stage without losing the good generalization performance”. What is the benefit or advantage for having faster early convergence without improving the final generalization performance? The authors claim that “M-SGD2 is more robust and achieves slightly better performance”, in Figure 1a, however, it is really hard to tell the difference between M-SGD2 and M-SGD from Figure 1a. The efficiency improvement in page 4 is really hard to follow for comparison with Algorithm 1 in page 3. Though m > 2 could reduce the number of operations in step 5 and 6, I don’t think this is a computational bottleneck. I believe these updates should be very fast in comparison with forward and gradient calculation. Making the 1% computation 50% faster does not mean more efficient. I would like to know how much computation cost can be saved with this modification. On the other hand, adding one more auxiliary buffer (scaled momentum) could significantly impact the training as the memory is often the limit. In section 3.1, what is “identical iteration”? It is hard to compare AM2-SGD with AM1-SGD. It would be easier to follow if the side-by-side algorithm comparison can be shown early. The section 4’s theoretical analysis based on the convex composite problem is not quite convincing. I am not sure how these results are related with the deep learning applications. In the experiment section, the comparison of AM1/2-SGD with other baselines seems not quite consistent. The authors first state that they use all 0.1 learning rate and 0.9 momentum for all methods, however, the setting for M-SGD is using 0.99 momentum and different learning rate schedule. This makes the comparison not very meaningful, while AM1-SGD and AM2-SGD do not use learning rate restart. With so many differences, the advantage of AM1-SGD and AM2-SGD are not that different with M-SGD. In the task of ImageNet-152, M-SGD even is better than AM1-SGD. This makes the conclusion that “AM1-SGD has a lightweight design, which can serve as an excellent replacement for M-SGD in large-scale deep learning tasks” not quite valid. Minor: The author may assume readers maybe familiar Katyusha momentum, I think there may need more background about it.
assistant
The authors proposed Amortized Nesterov’s Momentum, a variant of Nesterov’s momentum that utilizes several past iterates, instead of one iterate, to provide the momentum. The goal is to have more robust iterates, faster convergence in the early stage and higher efficiency. The authors designed two different realizations, AM1-SGD and AM2-SGD. Comments: My major concern for this paper is its unconvincing motivation and experiment results, especially when the approach is designed for deep learning. The motivation for the proposed approach is not quite convincing. The authors said that “due to the large stochasticity, SGD with Nesterov’s momentum is not robust...This increased uncertainty slows down its convergence especially in the early stage of training and makes its single run less trustworthy” For image classification, Nesterov momentum is very popular and the final convergence values of different trails seems to be similar. It would be more convincing if the authors can provide practical evidence for supporting this claim. It was claimed that Amortized Nesterov’s Momentum has “higher efficiency and faster convergence in the early stage without losing the good generalization performance”. What is the benefit or advantage for having faster early convergence without improving the final generalization performance? The authors claim that “M-SGD2 is more robust and achieves slightly better performance”, in Figure 1a, however, it is really hard to tell the difference between M-SGD2 and M-SGD from Figure 1a. The efficiency improvement in page 4 is really hard to follow for comparison with Algorithm 1 in page 3. Though m > 2 could reduce the number of operations in step 5 and 6, I don’t think this is a computational bottleneck. I believe these updates should be very fast in comparison with forward and gradient calculation. Making the 1% computation 50% faster does not mean more efficient. I would like to know how much computation cost can be saved with this modification. On the other hand, adding one more auxiliary buffer (scaled momentum) could significantly impact the training as the memory is often the limit. In section 3.1, what is “identical iteration”? It is hard to compare AM2-SGD with AM1-SGD. It would be easier to follow if the side-by-side algorithm comparison can be shown early. The section 4’s theoretical analysis based on the convex composite problem is not quite convincing. I am not sure how these results are related with the deep learning applications. In the experiment section, the comparison of AM1/2-SGD with other baselines seems not quite consistent. The authors first state that they use all 0.1 learning rate and 0.9 momentum for all methods, however, the setting for M-SGD is using 0.99 momentum and different learning rate schedule. This makes the comparison not very meaningful, while AM1-SGD and AM2-SGD do not use learning rate restart. With so many differences, the advantage of AM1-SGD and AM2-SGD are not that different with M-SGD. In the task of ImageNet-152, M-SGD even is better than AM1-SGD. This makes the conclusion that “AM1-SGD has a lightweight design, which can serve as an excellent replacement for M-SGD in large-scale deep learning tasks” not quite valid. Minor: The author may assume readers maybe familiar Katyusha momentum, I think there may need more background about it.