Aggregated Momentum: Stability Through Passive Damping
Momentum is a simple and widely used trick which allows gradient-based optimizers to pick up speed along low curvature directions. Its performance depends crucially on a damping coefficient. Largecamping coefficients can potentially deliver much larger speedups, but are prone to oscillations and instability; hence one typically resorts to small values such as 0.5 or 0.9. We propose Aggregated Momentum (AggMo), a variant of momentum which combines multiple velocity vectors with different damping coefficients. AggMo is trivial to implement, but significantly dampens oscillations, enabling it to remain stable even for aggressive damping coefficients such as 0.999. We reinterpret Nesterov*s accelerated gradient descent as a special case of AggMo and analyze rates of convergence for quadratic objectives. Empirically, we find that AggMo is a suitable drop-in replacement for other momentum methods, and frequently delivers faster convergence with little to no tuning.
 This paper proposed an aggregated momentum methods for gradient based optimization. The basic idea is instead of using a single velocity vector, multiple velocity vectors with different damping factors are used in order to improve the stability. In term of novelty, the proposed method seems quite incremental. Using multiple velocity vectors seems interesting but not surprising, There is no theoretical guideline how to determine the number of velocity vectors and how to choose the damping factors. I would also suggest that authors should put some main theoretical results like the convergence analysis to the main paper instead of the appendix. In terms of the clarity, I think the paper is well written and the experiments are sufficient and convincing. One minor question is: what is .lambda in Fig. 1?
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Aggregated Momentum: Stability Through Passive Damping
Momentum is a simple and widely used trick which allows gradient-based optimizers to pick up speed along low curvature directions. Its performance depends crucially on a damping coefficient. Largecamping coefficients can potentially deliver much larger speedups, but are prone to oscillations and instability; hence one typically resorts to small values such as 0.5 or 0.9. We propose Aggregated Momentum (AggMo), a variant of momentum which combines multiple velocity vectors with different damping coefficients. AggMo is trivial to implement, but significantly dampens oscillations, enabling it to remain stable even for aggressive damping coefficients such as 0.999. We reinterpret Nesterov*s accelerated gradient descent as a special case of AggMo and analyze rates of convergence for quadratic objectives. Empirically, we find that AggMo is a suitable drop-in replacement for other momentum methods, and frequently delivers faster convergence with little to no tuning.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper proposed an aggregated momentum methods for gradient based optimization. The basic idea is instead of using a single velocity vector, multiple velocity vectors with different damping factors are used in order to improve the stability. In term of novelty, the proposed method seems quite incremental. Using multiple velocity vectors seems interesting but not surprising, There is no theoretical guideline how to determine the number of velocity vectors and how to choose the damping factors. I would also suggest that authors should put some main theoretical results like the convergence analysis to the main paper instead of the appendix. In terms of the clarity, I think the paper is well written and the experiments are sufficient and convincing. One minor question is: what is .lambda in Fig. 1?
assistant
 This paper proposed an aggregated momentum methods for gradient based optimization. The basic idea is instead of using a single velocity vector, multiple velocity vectors with different damping factors are used in order to improve the stability. In term of novelty, the proposed method seems quite incremental. Using multiple velocity vectors seems interesting but not surprising, There is no theoretical guideline how to determine the number of velocity vectors and how to choose the damping factors. I would also suggest that authors should put some main theoretical results like the convergence analysis to the main paper instead of the appendix. In terms of the clarity, I think the paper is well written and the experiments are sufficient and convincing. One minor question is: what is .lambda in Fig. 1?