Preconditioned Spectral Descent for Deep Learning
Deep learning presents notorious computational challenges. These challenges include, but are not limited to, the non-convexity of learning objectives and estimating the quantities needed for optimization algorithms, such as gradients. While we do not address the non-convexity, we present an optimization solution that exploits the so far unused "geometry" in the objective function in order to best make use of the estimated gradients. Previous work attempted similar goals with preconditioned methods in the Euclidean space, such as L-BFGS, RMSprop, and ADAgrad. In stark contrast, our approach combines a non-Euclidean gradient method with preconditioning. We provide evidence that this combination more accurately captures the geometry of the objective function compared to prior work. We theoretically formalize our arguments and derive novel preconditioned non-Euclidean algorithms. The results are promising in both computational time and quality when applied to Restricted Boltzmann Machines, Feedforward Neural Nets, and Convolutional Neural Nets.
I find the explanation of the ADAspectral and RMSspectral interesting. A few comments that I have are: (1) Readability for wider audience:   - It might be useful (maybe in the supplementary material) to provide a  more verbose description of how one gets the update rule for either of  the methods. Or even better, code ! Most readers will first want to know  how hard it is to implement and how well it works before trying to  understand the math. From this perspective, algorithm 1 looks cryptic  with the #-operator and b-operator in there.    - I know it is said, but make it more clearer from the beginning that these  approach amounts to having a different update rule for the weights  (which are matrices) compared to biases. And that for the weights it  involves an SVD (or approximate SVD) over the gradient of these weights.  I think the paper will have a much bigger impact if it is really easy to  follow and clear from the start what the end result will be, even for one  that is not used to these things. (2) Intuitively one way of understanding this algorithm is the following(please correct me if I'm wrong). Say you set D to I (identity). Than youare simply using the sign of the gradient for the biases (multiply this bythe norm of the gradient), and projectingthe gradients of the weights on a sphere of radius equal to the largestsingular value. Now if D is different from I, it is almost like you rely onD completely to tell how much more you should move along an axis vsanother. Again for the biases this is simpler to see than for the weights. * First of all, it would be interesting to compare this with something likeRprop. Rprop basically keeps only the sign of the gradient and ignores itsmagnitude. The RMSspectral seems to project the gradients on some spherelike Rprop (just that you do a rescaling of the gradients with D beforethat and the radius of the sphere is not the learning rate, but the norm ofthe gradient), butthen to change the shape of the sphere based on D. * Secondly it would be interesting to know how vital is for D to be a goodapproximation of the curvature. E.g. it feels that if I make D to be I,I would get an algorithm that probably is much slower than SGD !? Is thisbecause now I don't have a tight bound on F(y). Is there some underlyingassumption on D in the derivation of the algorithm that can be easilymade visible (e.g. the bound becomes tighter if D becomes closer to thecurvature?). (3) I find it hard to compare the results the authors have with other resultsin the literature because they present the results in terms of normalizedtime (instead of actual time). It would have been useful to run the sameexperiment as the one of some other already published paper onoptimization and present the results in a similar fashion (e.g. the deepautoencoder on MNIST or something else).Minor things: * tiny spelling mistakes (e.g. line 403: epcohs)* figure 3, what does normalized time mean?