Linear Convergence with Condition Number Independent Access of Full Gradients
For smooth and strongly convex optimizations, the optimal iteration complexity of the gradient-based algorithm is O( √ κ log 1/ǫ), where κ is the condition number.
In the case that the optimization problem is ill-conditioned, we need to evaluate a large number of full gradients, which could be computationally expensive. In this paper, we propose to remove the dependence on the condition number by allowing the algorithm to access stochastic gradients of the objective function. To this end, we present a novel algorithm named Epoch Mixed Gradient Descent (EMGD) that is able to utilize two kinds of gradients. A distinctive step in EMGD is the mixed gradient descent, where we use a combination of the full and stochastic gradients to update the intermediate solution. Theoretical analysis shows that EMGD is able to find an ǫ-optimal solution by computing O(log 1/ǫ) full gradients and O(κ 2 log 1/ǫ) stochastic gradients.
quality: 7 (out of 10) clarity: 8 originality: 
      8 significance: 7 SUMMARY: The authors consider the problem of 
      optimizing a smooth and strongly convex function over a convex constraint 
      set such that the gradient mapping update can be computed efficiently. The 
      optimal first-order algorithm of Nesterov has linear convergence for such 
      problem but the constant depends on the square root of the condition 
      number k. The authors consider the situation where one has access to the 
      expensive full gradient of the objective as well as a cheap stochastic 
      gradient oracle. They propose a hybrid algorithm which only requires O(log 
      1/eps) calls to the full gradient oracle (independent of the condition 
      number) and O(k^2 log(1/eps)) calls to the cheaper stochastic gradient 
      oracle -- as long as the condition number is not too big, this could be 
      faster in theory. The main idea behind their algorithm(called Epoch Mixed 
      Gradient Descent - EMGD) is to replace a full gradient step (called an 
      epoch) with a fixed number O(k^2) of mixed gradient steps which use a 
      combination of the full gradient (computed once for the epoch) and 
      stochastic gradients (which vary within an epoch). By taking the average 
      of the O(k^2) iterates within an epoch, they can show a constant decrease 
      of the suboptimality *independent* of the condition number, which is why 
      the number of required full gradient step computations (the number of 
      epochs) is independent from the condition number. They provide a simple 
      and complete self-contained proof of their convergence rate, but no 
      experiment. I enjoyed reading this paper and I think that the idea 
      of mixing a few full gradient computations with a large number of cheap 
      stochastic gradient steps is novel and interesting. This work situates 
      itself in a string of recent papers which attempt to use the cheaper 
      stochastic oracle while maintaining a linear convergence rate. A recent 
      theoretical AND practical breakthrough was made with the SAG algorithm 
      [16] which works for a smooth strongly convex objective which is the sum 
      of n simple functions (such as in regularized empirical loss minimization 
      - where n is the number of training examples). In this case, a reasonable 
      assumption is that the full gradient oracle is n times more expensive to 
      compute than the stochastic gradient one. Then SAG is faster in theory to 
      Nesterov' algorithm as long as the condition number k < = n/8. In 
      contrast, EMGD in this case is faster to Nesterov' algorithm in theory as 
      long as the condition number k < = n^(2/3), and EMGD is slower than SAG 
      in all regimes (it has the same big O when k < = n^1/2). To get a sense 
      of these speed-ups, if k = n^1/2, then both SAG & EMGD are O(n 
      log(1/eps)) whereas Nesterov' algorithm O(n^(5/4) log(1/eps)). The two 
      main advantages that I see for EMGD over SAG are that 1) as mentioned by 
      the authors, EMGD works for constrained optimization [supposing that the 
      gradient mapping update can be computed efficiently] whereas SAG is only 
      defined so far for unconstrained problems; and 2) the convergence proof 
      for EMGD is much simpler than the one for SAG and so could yield more 
      insights as well as make the modifications to EMGD more amenable to 
      provable guarantees [their result is also stronger as it holds with high 
      probability vs. in expectation for SAG]. The authors also mention a 
      possible memory / parallelization advantage, though this is less clear as 
      SAG can also be parallelized using mini-batches (which also reduces the 
      memory requirement by the size of the mini-batches). EVALUATION 
      SUMMARY: Pros: - Gives a novel and interesting algorithmic idea 
      with a clean, simple and solid theory. - Like SAG, get a linear 
      convergence rate for regularized empirical loss minimization where the 
      condition number k is not multiplying the number of training examples n in 
      the constant; but their algorithm is more general (works for constrained 
      optimization) and the proof is much simpler. - The paper is clearly 
      written and the proof is self-contained. Cons: - There is no 
      experiment which could show that this algorithm could actually make a 
      difference in practice (and doesn't provide any concrete example to 
      illustrate why its suggested theoretical advantages could be relevant in 
      practice). - There is no discussion of the limitations / drawbacks of 
      the algorithm (especially, in comparison to the existing algorithms -- 
      section 3.4 should be improved! I make several suggestions in this 
      review). - The proof is lacking some high-level comments which could 
      justify the essential insights used to its construction. QUALITY: 
      The paper is technically sound. Some experiments would have been 
      appreciated, though I think that the theoretical contribution could stand 
      on its own. The authors should definitively extend section 3.4 with the 
      limitations and drawbacks of their algorithm though. They should also add 
      a more concrete discussion of the sum of n functions example which 
      highlights nicely the differences with SAG and Nesterov (as I mentioned 
      above -- e.g. EMGD is worse than Nesterov for (roughly) k > n^(2/3); 
      SAG is same as EMGD for 1 < = k < = n^(1/2); better than EMGD (and 
      Nesterov) for n^(1/2) < k < = n/8; and worse than Nesterov for k 
      > n/8). A major drawback of the EMGD for the practitioner is that the 
      number of steps within an epoch needs to be fixed in advance (with the 
      knowledge of k) -- in contrast, both SCDA [17] and SAG don't have to fix 
      the number of steps in advance and so can benefit for having a faster 
      practical convergence than the bound would predict (or benefit from better 
      local condition number e.g.). Moreover, SCDA has automatic step-size 
      selection; whereas SAG has an adaptive step-size heuristic which seems to 
      work quite well in practice. The authors should add this discussion in the 
      paper. ** [ADDENDUM after discussion with other reviewers: a) As 
      another reviewer mentioned, often in practice in ML the regularization 
      parameter is C/n and so the condition number is ~= n/C', which is a regime 
      in which SAG still does better than Nesterov, but in which EMGD *doesn't*. 
      This should be pointed out (and perhaps another practical setting where k 
      < n^(2/3) should mentioned). b) The authors should cite [Hybrid 
      Deterministic-Stochastic Methods for Data Fitting. M. Friedlander, M. 
      Schmidt. SISC, 2012] which also presents a hybrid deterministic-stochastic 
      algorithm with a *linear* rate of convergence. This latter algorithm still 
      has the condition number appearing in the rate though, so the ultimate 
      rate is not faster than standard gradient descent (just the beginning is 
      faster because of using cheaper steps) -- so the current submission can 
      still improve theoretically over this rate when k is not too big with 
      respect to n.] ** CLARITY: The paper is fairly clear. I have 
      appreciated the summary from Table 1 which I haven't seen in the 
      literature in such a clear manner. The proof can be followed tightly, but 
      would be more useful to the reader if the authors could add a few 
      high-level comments motivating some of the defined quantities which are 
      fitting together a bit too magically in its current form. 
      ORIGINALITY: This is a novel combination of known techniques. 
      SIGNIFICANCE: The practical relevance of the algorithm is not 
      demonstrated yet. But the theoretical contribution could have impact. In 
      the context of the difficult proof of convergence of SAG (and the simple 
      proof of SDCA, but which only applies to a restricted setup), the simple 
      proof of EMGD is a major contribution. == More detailed 
      suggestions == - line 050: I suggest to say "are summarized in 
      Table 1 and detailed in the related work section". One can wonder at first 
      what are the citations for these rates. - line 118-119: I would 
      specify here that SCDA is only defined for a specific form of f_i, unlike 
      SAG which handles any smooth convex f_i. - line 132: I would 
      explicitly state that this condition is "for all w" -- it was a bit 
      ambiguous whether the condition was only for a fixed w (the 'given input 
      point w' mentioned in 1), which of course would not be sufficient to have 
      the gradients match. - line 139: I would mention explicitly that 
      we also have \grad F(w) = E[\grad f(w) ]. - line 224: The claim 
      that SAG (or SCDA) cannot take advantage of distributed computing is 
      false: they both can use mini-batches (this can also reduce the memory 
      requirement). - line 288: I would write "(7) on (10) with x=w* 
      (feasible by (5)) and x*=w_{t+1}" to be more explicit -- it is not that 
      obvious how to obtain the line otherwise... - line 304 (and all 
      other places): Add parenthesis around (F(w_t)-F(w*)) to be explicit that 
      F(w*) is also summed T times... - line 366: Add that the 2nd 
      inequality is only valid for all T > = 1 if delta < = exp(-1/2) 
      [this also explains a condition stated in (4) which appeared nowhere 
      explicitly]. Some typos: -042: 'an convex' - > 'a convex' 
      -056: *strong* convexity, *condition* number -088 (and other 
      places): please correct the notation of your domain to b consistent -- you 
      use D in 039 and the first few pages; here, you use \Omega -- choose one 
      and use it everywhere in the paper! - 307: use norm symbol instead of 
      absolute value -366: for consistency, replace log with ln in the 
      middle equation === Update after rebuttal === I am 
      happy with the response by the authors, but note that I will carefully 
      check that their updated version is implementing the changes that we have 
      suggested!