AGGREGATED MOMENTUM: STABILITY THROUGH PASSIVE DAMPING
Momentum is a simple and widely used trick which allows gradient-based optimizers to pick up speed along low curvature directions. Its performance depends crucially on a damping coefficient β. Large β values can potentially deliver much larger speedups, but are prone to oscillations and instability; hence one typically resorts to small values such as 0.5 or 0.9. We propose Aggregated Momentum (AggMo), a variant of momentum which combines multiple velocity vectors with different β parameters. AggMo is trivial to implement, but significantly dampens oscillations, enabling it to remain stable even for aggressive β values such as 0.999. We reinterpret Nesterov's accelerated gradient descent as a special case of AggMo and analyze rates of convergence for quadratic objectives. Empirically, we find that AggMo is a suitable drop-in replacement for other momentum methods, and frequently delivers faster convergence with little to no tuning.
The paper introduces a variant of momentum that aggregates several velocities with different dampening coefficients. The proposed optimization algorithm can significantly decrease oscillation thus one can use much larger dampening coefficient to achieve faster convergence.
The paper does a good job of motivating the algorithm, both in terms of intuitive reasoning from physics, and some demonstrative examples. The paper also has nice analysis in the simple quadratic case where it tries to make equivalence to the Nesterov's accelerated gradients.
The experiments are also thorough and convincing since it contains various network architectures and different datasets. From the comparison, it seems AggMo consistently achieves the best or comparable performance even in test error/accuracies.
The connection to Nesterov's accelerated gradient and extragradient methods can be discussed in more details. It'll be nice to put the theoretical results in the main text as well.