We recall that if A is an invertible and symmetric real m\times m matrix, then it is diagonalisable. Therefore, if we denote by \mathcal{E^{+}(A)\subset R^m (respectively E^{-}(A)\subset R^m) to be the vector subspace generated by eigenvectors with positive eigenvalues of A (correspondingly the vector subspace generated by eigenvectors with negative eigenvalues of A), then we have an orthogonal decomposition R^m=E^{+}(A)\oplus E^{-}(A). Hence, every x\in R^m can be written uniquely as x=pr_{A,+}(x)+pr_{A,-}(x) with pr_{A,+}(x)\in E^{+}(A) and pr_{A,-}(x)\in E^{-}(A). We propose the following simple new modification of quasi-Newton's methods. }%DIFDELCMD < {\bf %%%  New Q-Newton's method. Let \Delta = \{\delta _0,\delta _1,\delta _2,\ldots \} be a countable set of real numbers which has at least m+1 elements. Let f:R^m\rightarrow R be a C^2 function. Let \alpha >0. For each x\in \mathbb{R^m such that \nabla f(x)}%DIFDELCMD < \not%%%  = 0, let \delta (x) = \delta _j, where j is the smallest number so that \nabla ^2f(x)+\delta _j||\nabla f(x)||^{1+\alphaId is invertible. (If \nabla f(x)=0, then we choose \delta (x)=\delta _0.) Let x_0\in \mathbb{R}^m be an initial point. We define a sequence of x_n\in \mathbb{R}^m and invertible and symmetric m\times m matrices A_nas follows: A_n=} \nabla ^2f(x_n)+\delta (x_n) ||\nabla f(x_n)|| ^{1+\alpha = x_n-w_n, where w_n=pr_{A_n,+ A_n^{-1}  \nabla f(x_n).  The main result of this paper roughly says that if f is C^3 and a sequence \{x_n\}, constructed by the New Q-Newton's method from a random initial point x_0, {\bf converges}, then the limit point is  not a saddle point, and the convergence rate is the same as that of Newton's method  .
We ^{+}(A)\subset R^m (respectively E^{-}(A)\subset R^m) to be the vector subspace generated by eigenvectors with positive eigenvalues of A (correspondingly the vector subspace generated by eigenvectors with negative eigenvalues of A), then we have an orthogonal decomposition R^m=E^{+}(A)\oplus E^{-}(A). Hence, every x\in R^m can be written uniquely as x=pr_{A,+}(x)+pr_{A,-}(x) with pr_{A,+}(x)\in E^{+}(A) and pr_{A,-}(x)\in E^{-}(A). We propose the following simple new modification of quasi-Newton's methods. }%DIFDELCMD < {\bf %%% propose in this paper New Q-Newton's method. The update rule for the simplest version is x_{n+1 = ^m such that \nabla f(x)}%DIFDELCMD < \not%%% x_n-w_n where w_n = pr_{A_n,+ = Id is invertible. (If \nabla f(x)=0, then we choose \delta (x)=\delta _0.) Let x_0\in \mathbb{R}^m be an initial point. We define a sequence of x_n\in \mathbb{R}^m and invertible and symmetric m\times m matrices A_nas follows: A_n=} \nabla ^2f(x_n)+\delta _n ||\nabla f(x_n)|| ^2 .Id and v_n = A_n^{-1} . \nabla f(x_n). Here \delta _n is an appropriate real number so that A_n is invertible, and pr_{A_n,\pm The main result of this paper roughly says that if f is C^3 and a sequence \{x_n\}, constructed by the New Q-Newton's method from a random initial point x_0, {\bf converges}, then the limit point is a critical point and is not a saddle point, and the convergence rate is the same as that of Newton's method . At the end of the paper, we present some issues (saddle points and convergence) one faces when implementing Newton's method and modifications into Deep Neural Networks. In the appendix, we test the good performance of New Q-Newton's method on various benchmark test functions such as Rastrigin, Askley, Rosenbroch and many other, against algorithms such as Newton's method, BFGS, Adaptive Cubic Regularization, Random damping Newton's method and Inertial Newton's method, as well as Unbounded Two-way Backtracking Gradient Descent. The experiments demonstrate in particular that the assumption that f is C^3 is necessary for some conclusions in the main theoretical results .
D
recall that if A is an invertible and symmetric real m\times m matrix, then it is diagonalisable. Therefore, if we denote by \mathcal{E
A
propose in this paper
R
Let \Delta
The update rule for the simplest version is x_{n+1
D
\{\delta _0,\delta _1,\delta _2,\ldots \} be a countable set of real numbers which has at least m+1 elements. Let f:R^m\rightarrow R be a C^2 function. Let \alpha >0. For each x\in \mathbb{R
A
x_n-w_n where w_n
R
0, let \delta (x)
pr_{A_n,+
D
\delta _j, where j is the smallest number so that \nabla ^2f(x)+\delta _j||\nabla f(x)||^{1+\alpha
R
(x_n)
_n
R
^{1+\alpha
^2 .Id and v_n
D
x_n-w_n, where w_n=pr_{A_n,+
A
.
A
Here \delta _n is an appropriate real number so that A_n is invertible, and pr_{A_n,\pm
A
a critical point and is
A
. At the end of the paper, we present some issues (saddle points and convergence) one faces when implementing Newton's method and modifications into Deep Neural Networks. In the appendix, we test the good performance of New Q-Newton's method on various benchmark test functions such as Rastrigin, Askley, Rosenbroch and many other, against algorithms such as Newton's method, BFGS, Adaptive Cubic Regularization, Random damping Newton's method and Inertial Newton's method, as well as Unbounded Two-way Backtracking Gradient Descent. The experiments demonstrate in particular that the assumption that f is C^3 is necessary for some conclusions in the main theoretical results