How To Make the Gradients Small Stochastically: Even Faster Convex and Nonconvex SGD *
Stochastic gradient descent (SGD) gives an optimal convergence rate when minimizing convex stochastic objectives f (x). However, in terms of making the gradients small, the original SGD does not give an optimal rate, even when f (x) is convex. If f (x) is convex, to find a point with gradient norm ε, we design an algorithm SGD3 with a near-optimal rate O(ε −2 ), improving the best known rate O(ε −8/3 ) of [17]. If f (x) is nonconvex, to find its ε-approximate local minimum, we design an algorithm SGD5 with rate O(ε −3.5 ), where previously SGD variants only , 14, 30]. This is no slower than the best known stochastic version of Newton's method in all parameter regimes [27].
This work studies convergence rates of the gradients for convex composite objectives by combining Nesterov’s tricks used for gradient descent with SGD. The authors provide three approaches which differ from each other only slightly and they provide the convergence rates for all the proposed approaches. My comments on this work are as follow:   1. It is indeed important to study convergence rates of gradients especially for non-convex problems. The authors motivate the readers by mentioning this but they assume convexity in their problem set-up. It would have been fine if they were comparing other convergence rates for convex problems as well but I am not sure the existing results they discussed are defined for convex. For instance the work in citation 11 discusses the rates for nonconvex functions. Also, I could not locate the rate \epsilon^{-8/3} in citation 11. I’ll appreciate if the authors clearly mention which result and what assumptions they used from citation 11 in their rebuttal. Same concern applies to the results in citation 15.  2. My general objection about the paper is its representation and organization. I found it really hard to follow the main motivation from a stack of lemmas, corollaries and theorems. I’d suggest authors to narrow their focus to maybe one approach but discuss it in detail. Abstract should also be made more clear. It is not obvious that the manuscript only focuses on convex problems just by reading the abstract. A section of a discussion or a conclusion would also make the manuscript more readable.  3. What is the relationship between \epsilon and \delta between lines 41 and 42?  4. I don’t really understand what “replacing the use of inequality” means in lines 50-51.  5. I am not sure the statement in line 113 about “online” is correct. I understand the term “online” could be used for different things in machine learning but I think it is important to clarify what authors really mean by online setting in this work. Do they mean batch training which requires the entire training data or training with data which appears in a sequential order.  Overall, although the work looks interesting in terms of some theoretical convergence results, I think there is still some room to make the manuscript more convincing by working on the representation.