Convergence of Cubic Regularization for Nonconvex Optimization under KŁ Property
Cubic-regularized Newton's method (CR) is a popular algorithm that guarantees to produce a second-order stationary solution for solving nonconvex optimization problems. However, existing understandings of the convergence rate of CR are conditioned on special types of geometrical properties of the objective function. In this paper, we explore the asymptotic convergence rate of CR by exploiting the ubiquitous Kurdyka-Łojasiewicz (KŁ) property of nonconvex objective functions. In specific, we characterize the asymptotic convergence rate of various types of optimality measures for CR including function value gap, variable distance gap, gradient norm and least eigenvalue of the Hessian matrix. Our results fully characterize the diverse convergence behaviors of these optimality measures in the full parameter regime of the KŁ property. Moreover, we show that the obtained asymptotic convergence rates of CR are order-wise faster than those of first-order gradient descent algorithms under the KŁ property.
The paper investigates the convergence of different measurement of cubic regularization method for non-convex optimization under KL property. It consists with a list of work on CR methods based on the analysis of Nesterove el.s' work. Since the type of methods can guarantee the convergence to the second-order stationary point, it is quite popular also considering the raising of training neural networks.   The paper is well-written, clear-organized and the theorems and proofs are easy to follow. Note that this is a pure theoretical work i.e., without new algorithms and/or numerical experiments. I have the following concerns  1. The results of the paper is based on KL property, and show convergence result in terms of the critical parameter \theta of KL. As a generalized function property, it is not easy to have \theta for a given function and the paper does not provide examples where this \theta can be values other than  \theta=1/2 for some machine learning problems.  2. To apply the algorithm, the Lipschitz-Hessian constant and an exact solution for subproblem is needed to derive the second-order stationary point convergence, which makes the algorithm unpractical for large problems. 3. There is no discussion over finite sum problems, which is typical in machine learning field, which I believe NIPS has more focus on such problems.  Considering the above, the paper is good but might not suitable for NIPS. I would therefore only vote the paper a weak accept but I'm also open to others suggestions.   minors: 1. \bar{x} in Definition 1 is not used and make the definition confusing.