A Universal Catalyst for First-Order Optimization
We introduce a generic scheme for accelerating first-order optimization methods in the sense of Nesterov, which builds upon a new analysis of the accelerated proximal point algorithm. Our approach consists of minimizing a convex objective by approximately solving a sequence of well-chosen auxiliary problems, leading to faster convergence. This strategy applies to a large class of algorithms, including gradient descent, block coordinate descent, SAG, SAGA, SDCA, SVRG, Finito/MISO, and their proximal variants. For all of these methods, we provide acceleration and explicit support for non-strongly convex objectives. In addition to theoretical speed-up, we also show that acceleration is useful in practice, especially for ill-conditioned problems where we measure significant improvements.
The authors propose a general scheme of wrapping an existing first order gradient method to solve the minimization of the original objective plus an iteratively sharper proximal term. The authors show that for a generic first order method this 'accelerates' the convergence by changes the depends on \frac{L}{\mu} to \sqrt{\frac{L}{\mu}}.Overall, it is a goodpaper with a couple deficiencies in(a) experiments and(b) given the existing algorithms, when exactly would a catalyst be useful ? (especially in ML). Despite the deficiencies, I still vote for acceptance as it has some ideas which could spur future research, for e.g. the idea of doing multiple passes over data with each pass having a sharper proximal term (similar to [8])CommentsPage 2, line 57 - are the authors sure that 'all' the incremental methods have a O(n) cost ? Including ProxSVRG ?The authors need to make it clear when their contributions are useful. For e.g., - when users cannot use the dual, and - ideally when the objective is not strongly convex (saying that n < L/mu seems, in my opinion, is a little weak and less useful regime)- and when data cannot be held in memory (as the inner algorithm can make a sequential passes over data).line 314, redundant 'do'line 390 - what does 'default' parameters ? default as in optimal w.r.t theory ? or just happen to be parameters in some source code ? The authors of SAG also outlined a procedure to do learning rate scheduling - was this taken into account.Was 'L' just set to the upperbound 'L' in the dataset ? was there any effort to tune this ?How was \eta set ? Was there any effort taken to ensure that enough tuning was performed on all the algorithms ? or was it the case that the authors picked a set of values and lucked out by getting a better performance on the proposed scheme ?One of the benefits of using a method like SAG/SAGA/SVRG/.. is to ensure quick initial convergence which is not guaranteed by FG or Acc-FG. It would be nice to see how the proposed method does w.r.t Acc-FG.SuggestionsTheorem 3.3. is misleading at a first glance, could the authors please ensure the contents of line line 256-257 are located in a more accessible place ? like the theorem statement itself.Table 1 caption " To simplify, we only present the case where n <= L/\mu when \mu > 0." could be boldened to ensure that it is read