Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
A central challenge to many fields of science and engineering involves minimizing non-convex error functions over continuous, high dimensional spaces. Gradient descent or quasi-Newton methods are almost ubiquitously used to perform such minimizations, and it is often thought that a main source of difficulty for these local methods to find the global minimum is the proliferation of local minima with much higher error than the global minimum. Here we argue, based on results from statistical physics, random matrix theory, neural network theory, and empirical evidence, that a deeper and more profound difficulty originates from the proliferation of saddle points, not local minima, especially in high dimensional problems of practical interest. Such saddle points are surrounded by high error plateaus that can dramatically slow down learning, and give the illusory impression of the existence of a local minimum. Motivated by these arguments, we propose a new approach to second-order optimization, the saddle-free Newton method, that can rapidly escape high dimensional saddle points, unlike gradient descent and quasi-Newton methods. We apply this algorithm to deep or recurrent neural network training, and provide numerical evidence for its superior optimization performance.
---------------------Basically, I think it's really important to show that |H| is better than the Fisher/Natural Gradient matrix in an "apples to apples" comparison.  Same setup, same learning rate heuristics, only different curvature.   It's important because the Fisher matrix is known to be much better for nonlinear optimization than the Hessian.  The experiment that you mention show that |H| is also much better than the Hessian.  So the big question is, which is better:  |H| or F, where both matrices are used in the same setup.---------------------Optimization methods assume that they are given convex approximations to the objective function.  However, this assumption is clearly false due to the presence of the large number of saddle points in high dimensional spaces.  The idea of explicitly modelling and dealing with saddle points is very interesting, and the results are intriguing:  working with the absolute value of the eigenvalues really does seem to result in better performance on the tasks considered.However, I feel that the experiments do not quite prove the point of the paper as well as it could have.  Specifically,  the experiments do not show convincingly that working with the absolute value of the Hessian is better than the Natural Gradient, or the Gauss Newton.  To be precise, I noticed that Saddle Free Newton was initialized by SGD.  In contrast, previous work on second order methods, including Martens (2010), did not initialize from SGD.  And while you report better results, it is possible that most of the improvement was caused by the use of SGD as a “pretraining” stage of the second order.  Thus the paper would be much more convincing if alongside SFN it included the same experiment (with the same optimizer, Krylov Subspace Descent), with the Gauss-Newton matrix.   At present, there is only a comparison with damped newton. The second issue, which is small yet significant, is that too much space and emphasis was placed on unnecessary formal justification of the method, introducing a new trust region method (which isn’t new --- the classical derivation of natural gradients minimizes the linear gradient over a quadratic constraint, which is precisely equivalent to an elliptical trust region method).  Likewise, the |H| matrix has positive eigenvalues, and therefore dividing by this positive definite matrix is a proper optimization procedure.  I strongly recommend the authors to replace the Hessian with the Gauss-Newton / Natural Gradient matrix while keeping everything else fixed (and spending the appropriate amount of time tuning the new setting), in order to see whether it is the Hessian that contributes to the better performance, or whether it is the SGD pre-training.