A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights
We derive a second-order ordinary differential equation (ODE), which is the limit of Nesterov's accelerated gradient method. This ODE exhibits approximate equivalence to Nesterov's scheme and thus can serve as a tool for analysis. We show that the continuous time ODE allows for a better understanding of Nesterov's scheme. As a byproduct, we obtain a family of schemes with similar convergence rates. The ODE interpretation also suggests restarting Nesterov's scheme leading to an algorithm, which can be rigorously proven to converge at a linear rate whenever the objective is strongly convex.
==========added after author rebuttal============Re Theorem 4.3. I think my initial comment is correct but maybe I should have been more explicit: Eq. (36) in Tseng's paper or the similar stepsize in Beck and Teboulle need not be satisfied with equality. When one bounds the suboptimality wrt a global minimizer, such as the f* in the author's Theorem 4.3, the inequality <= in Tseng's Eq. (36) is enough, which makes the stepsize (r-1)/(k+r-1) feasible. The strict equality is needed when one compares wrt an arbitrary point, such as in the theorem Tseng derived. This is apparent if one checks when Eq. (36) is used in the proof, or when Beck and Teboulle applied their stepsize rule (proof of their Lemma 4.1, 2nd displayed equation in page 195, note that v_k >=0). Re Experiments: The authors seem to misunderstand my comment. I was not asking for a thorough comparison with a great number of competitive algorithms, but a few that naturally pop out in one's mind. These comparisons not only give us a better picture but also provide guidance for others who might be interested in applying the proposed algorithms. ======================================Nesterov's accelerated gradient algorithm has been a very popular algorithm for handling large datasets, and is known to be optimal among first order methods. The authors re-derived Nestorov's algorithm as a discretization of a second order differentiable equation. It appeared the convergence of the continuous ODE is much cleaner to deal with. Moreover, inspired by the continuous ODE, the authors proposed two variations, one of which could exploit strong convexity even when the parameter is not known beforehand. Limited experimental results partially validate the theoretical discoveries. As far as the reviewer is concerned, the convergence results derived in this submission are novel and potentially significant. Although the idea to relate continuous ODE with discrete gradient methods has a long history, the authors have done a superb job in instantiating the general principle to Nesterov's algorithm, overcoming some technical difficulties along the way. Theorem 4.3 is not quite as new as it appears to be. In fact, the update rule in Eq. (12) makes it clear that it is nothing but the same update rule in Eq (34) in the following paper:"Approximation accuracy, gradient methods, and error bound for structured convex optimization, Tseng'10", under the identification theta_k = (r-1) / (k+r-1). In fact, Tseng's treatment is even (slightly) more general, with a very enjoyable proof too. This should be pointed out. The equation in line 260 seems to be new and interesting.Theorem 4.4 is very interesting as it does not require the knowledge of the strong convexity parameter. Unfortunately the current proof appears to be long and tedious. It would be great to polish the proof to further distill the essence. On the other hand, in many machine learning applications the strong convexity indeed is known beforehand (in particular if one uses strongly convex regularizers). The current experiments are not very satisfying. Two very natural comparisons are missing (when strong convexity is present): 1). the usual proximal gradient (without accelerating), which achieves a linear rate with a worse constant; 2). Nesterov's algorithm with the knowledge of the strong convexity parameter. These comparisons will give us a better idea on how effective the restarting strategy is. The experiment suffers also from another apparent issue: Nesterov's original algorithm's oscillating behavior. It is trivial to fix this issue by enforcing monotonicity, i.e., picking y_t as the best (in terms of objective value) of y_t and y_{t-1}. The convergence property will not he jeopardized (because the momentum term is still changing). This is a third competitor the authors should have compared to. Although the authors did give some justification of the restarting strategy, the results are not as convincing as it could be at the moment: 1). Theorem 5.2 is about the continuous ODE and does not allow any nonsmooth component; 2). How does it compare to a simple halving trick, i.e., one starts with a guess mu of the strong convexity parameter and keeps halving it?