Adaptive Negative Curvature Descent with Applications in Non-convex Optimization
Negative curvature descent (NCD) method has been utilized to design deterministic or stochastic algorithms for non-convex optimization aiming at finding second-order stationary points or local minima. In existing studies, NCD needs to approximate the smallest eigen-value of the Hessian matrix with a sufficient precision (e.g., 1) in order to achieve a sufficiently accurate second-order stationary solution (i.e., λ min (∇ 2 f (x)) ≥ − 2 ). One issue with this approach is that the target precision 2 is usually set to be very small in order to find a high quality solution, which increases the complexity for computing a negative curvature. To address this issue, we propose an adaptive NCD to allow an adaptive error dependent on the current gradient's magnitude in approximating the smallest eigen-value of the Hessian, and to encourage competition between a noisy NCD step and gradient descent step. We consider the applications of the proposed adaptive NCD for both deterministic and stochastic non-convex optimization, and demonstrate that it can help reduce the the overall complexity in computing the negative curvatures during the course of optimization without sacrificing the iteration complexity.
This paper utilizes the Negative curvature descent (NCD) method to design adaptive algorithms for non-convex optimization aiming at finding second-order stationary points or local minima. The theoretical convergence rate of the adaptive NCD matches the state-of-the-art NCD-AG. Supporting experiment results using cubic regularization, regularized non-linear least-square, and one hidden-layer NN implies a significant supercedes the NCD-AG.  In some sense, this work is a continuation of the work by Carmon et al (2016) in the deterministic setting. This work made a key contribution that at each update of AdaNCD\textsuperscript{det}, it chooses one of the GD (gradient-descent) step and NCD (Negative curvature descent / Lanczos) step, whichever decreases the function value more. The benefit of this method is that one only need to solve NCD with an accuracy of $O(\sqrt{\epsilon} \lor \|\nabla f(x)\|^{1/2})$ instead of $O(\sqrt{\epsilon})$. It is unsurprising to me (and also argued in the paper) that the convergence rate matches the state-of-the-art $O(\epsilon^{-1.75})$ gradient complexity when finding an $(\epsilon,\sqrt{\epsilon})$-approximate SOSP, which is the state-of-the-art local-minimizer finding algorithm [Carmon et al 2016, Agarwal et al 2017].  I tend to vote yes for this paper, not because of its technical proof method which has largely appeared in earlier literatures, but because the simple adaptive twist that makes it numerically work in practice. Most non-convex optimization theory works notoriously lack of such numerical support, however, making them numerically unattended.  The work also dealt with the case when $\epsilon_2 = \epsilon^\alpha$ with $\alpha\in (0,1)$, and stochastic setting. Perhaps, in future work, the author also want to include the analysis of the finite-sum setting.   Some minor comments:  In Algorithm 5, the authors might want to explain why one can solve AdaNCG at accuracy $\epsilon^{3/4}$ and then switch gear to the AGD iteration. Should the accuracy be $\epsilon$ so they match up?  The authors might want to carefully name the algorithms so as not to cause any confusion. For example, Algorithm 1, 3 and 5 have similar names, but the second (resp.~third) serves as a subroutine of the first (resp.~second).  For the stochastic setting, the AdaNCD+SCSG result seems very similar to the so-called Neon+SCSG method (Xu et al 2017; Allen Zhu and Li, 2017). Does the complexity in Theorem 4 matches up with Neon2 in Allen-Zhu and Li (2017)?