Stochastic Nested Variance Reduction for Nonconvex Optimization
We study finite-sum nonconvex optimization problems, where the objective function is an average of n nonconvex functions. We propose a new stochastic gradient descent algorithm based on nested variance reduction. Compared with conventional stochastic variance reduced gradient (SVRG) algorithm that uses two reference points to construct a semi-stochastic gradient with diminishing variance in each iteration, our algorithm uses K + 1 nested reference points to build a semi-stochastic gradient to further reduce its variance in each iteration. For smooth nonconvex functions, the proposed algorithm converges to an ✏-approximate first-order stationary point (i.e., krF (x)k 2  ✏) within e O(n^✏ 2 + ✏ 3^n1/2 ✏ 2 ) 1 number of stochastic gradient evaluations. This improves the best known gradient complexity of SVRG O(n + n 2/3 ✏ 2 ) and that of SCSG O(n^✏ 2 + ✏ 10/3^n2/3 ✏ 2 ). For gradient dominated functions, our algorithm also achieves better gradient complexity than the state-of-the-art algorithms. Thorough experimental results on different nonconvex optimization problems back up our theory.
The paper proposes a stochastic nested variance reduced gradient descent method for non-convex finite-sum optimization. It has been studied that variance reduction in stochastic gradient evaluations improves the complexity of stochastic gradient evaluations. A popular method is stochastic variance reduced gradient (SVRG), which uses a single reference point to evaluate the gradient. Inspired by this, authors introduce variance reduction using multiple reference points with nested scheme. More precisely, each reference point updates in every T steps and the proposed algorithm uses K points and hence one-epoch iterates T^K loops. Authors demonstrate that gradient complexity of their algorithm is better than the original SVRG and stochastically controlled stochastic gradient (SCSG) under mild conditions. They also provide an algorithm and analysis for gradient dominated case, where the global optimal can be found. Finally, their algorithms show better results for practical nonconvex problems.  Although the idea of this paper seems to be natural, its result is very important in the sense of good theoretical analysis and experimental justifications. This paper is well written and easy to read. Hence, I believe that this paper should be accepted in NIPS.  One confusion is that it would be better if authors provide running time plot in experiments. Some readers may not satisfy with epochs-error plots. It makes this work to be more strengthen.   ***** I have read the author feedback. I appreciate authors for providing the details about my questions. *****