First-order Stochastic Algorithms for Escaping From Saddle Points in Almost Linear Time
In this paper, we consider first-order methods for solving stochastic non-convex optimization problems. The key building block of the proposed algorithms is firstorder procedures to extract negative curvature from the Hessian matrix through a principled sequence starting from noise, which are referred to NEgative-curvatureOriginated-from-Noise or NEON and are of independent interest. Based on this building block, we design purely first-order stochastic algorithms for escaping from non-degenerate saddle points with a much better time complexity (almost linear time in the problem's dimensionality) under a bounded variance condition of stochastic gradients than previous first-order stochastic algorithms. In particular, we develop a general framework of first-order stochastic algorithms with a secondorder convergence guarantee based on our new technique and existing algorithms that may only converge to a first-order stationary point. For finding a nearly second-order stationary point x such that ∇F (x) ≤ and ∇ 2 F (x) ≥ − √ I (in high probability), the best time complexity of the presented algorithms is O(d/ 3.5 ), where F (·) denotes the objective function and d is the dimensionality of the problem. To the best of our knowledge, this is the first theoretical result of first-order stochastic algorithms with an almost linear time in terms of problem's dimensionality for finding second-order stationary points, which is even competitive with existing stochastic algorithms hinging on the second-order information.
This paper proposes an efficient algorithm, Neon, to extract the negative-curvature of the Hessian matrix. Neon is very useful, which can be taken as a build block to convert existing convert existing stationary point finding algorithms into local minimum finding algorithms. This is because although the gradient of the points around saddle point is very small, there still exists a direction to decrease the function value by updating the variable along the negative-curvature of its Hessian matrix. In this way, Neon can be easily integrated with existing stationary point finding algorithms, such as SGD, accelerated SGD, SCGC, etc. Actually, the authors theoretically showed that SGD, SCGC, Natasha and SVRG can be combined with Neon and compared with existing works, such as noisy SGD and SGLD, the achieved computational complexity has much better dependence on the dimension d of the problem. Specifically, the complexity of noisy and SGLD rely on O(d^p) (p>=4), while the complexity in this work is only O(d). For the optimization accuracy, the best complexity in this work makes about improvement O(1/epsilon^0.75). Compared with Hessian vector product based algorithm, Neon only need to use first order information and is thus more practical.   To my best knowledge, this work seems to be the first work which firstly use first-order gradient to compute the negative-curvature for escaping saddle points. Based on this work, there are some follow-up/independent works, such as Neon2 proposed by Allen-Zhu et al.  So I think that this paper has very strong novelty and makes great contributions to optimization area, especially for escaping saddle points.    However, the discussion of related works is not sufficient. For example, the differences between Neon and Neon2, AGD proposed by Jin Chi are not very clear, since they share similar techniques and the authors only slightly mentioned the differences.   The authors claim that their work is the first one which only linearly depends on the problem dimension d. I wonder whether existing works, such as Natasha2, SNCG although they involve the second information, e.g. Hessian vector product, also depend on the problem dimension d polynomially or higher.  Finally, as for the experiments, the authors only show the convergence behavior of Neon-SGD, Neon+-SGD and noisy SGD in Fig. 2. However, these plots only demonstrated that these algorithms can converge but cannot prove they can escape saddle points. A better comparison may be that adding an algorithm, e.g. SGD, and looking at SGD can achieve the same optimization accuracy with Neon-SGD, Neon+-SGD and noisy SGD.