Exact natural gradient in deep linear networks and application to the nonlinear case
Stochastic gradient descent (SGD) remains the method of choice for deep learning, despite the limitations arising for ill-behaved objective functions. In cases where it could be estimated, the natural gradient has proven very effective at mitigating the catastrophic effects of pathological curvature in the objective function, but little is known theoretically about its convergence properties, and it has yet to find a practical implementation that would scale to very deep and large networks. Here, we derive an exact expression for the natural gradient in deep linear networks, which exhibit pathological curvature similar to the nonlinear case. We provide for the first time an analytical solution for its convergence rate, showing that the loss decreases exponentially to the global minimum in parameter space. Our expression for the natural gradient is surprisingly simple, computationally tractable, and explains why some approximations proposed previously work well in practice. This opens new avenues for approximating the natural gradient in the nonlinear case, and we show in preliminary experiments that our online natural gradient descent outperforms SGD on MNIST autoencoding while sharing its computational simplicity.
Update: I have considered the author response. I maintain my assessment.  ==========================================================  This paper proposed a convergence analysis on natural gradients. The authors show that the natural gradient updates converge exponentially fast for the deep linear networks with no activation functions and no bias terms. To make the updates faster, the authors proposed some approximation techniques as well as an online implementation. The authors also analyzed the Fisher information structure for the deep linear networks and argue that the block-diagonal style approximations [2, 3, 4, 5, 6] to the inverse Fisher information matrix may ignore the effects of the off-diagonal blocks during the updates. Meanwhile, through analyzing the deep linear networks, the authors explained that the off-diagonal terms contribute the same as the diagonal terms in the updates, and hence explain why block-diagonal approximations may work well in practice.  The problem addressed by this paper is quite interesting. Analyzing the convergence rates of natural gradient methods for general deep neural networks is an open problem, to best of my knowledge. Also, the analysis on the block-diagonal approximation style methods is interesting since these methods are widely applied by the machine learning community. However, I am not sure if the convergence rate analysis of the deep linear networks is useful or not since these networks may not be useful in practice while the case of general deep neural networks may be quite different with this special case.  Strengths of the paper:  1. In my opinion, the research direction of studying the convergence rates of natural gradient updates is interesting and useful, since natural gradients are widely used.  2. Using the deep linear networks to analyze the off-diagonal blocks' contributions on natural gradient updates is a very good idea. This analysis helps us understand why block-diagonal style approximations could work.  Weakness:  1. Though the analysis on deep linear networks show us some insights on the convergences rates of natural gradient updates, the case might be very different for general deep neural networks since they contain bias terms and non-linear activation functions, which might make things much more complicated.  2.  For the deep linear networks with layer sizes n_0, ...., n_L, only the case when n_0 * n_L > \sum\limits_{l=0}^{L-1} n_l * n_{l + 1} is useful (otherwise, it is better just to apply a single matrix W, but not a deep linear network). However, for both experiments in Figure 1(A) and Figure 1(B), \sum\limits_{l=0}^{L-1} n_l * n_{l + 1} is much larger than n_0 * n_L. It is better to show more results on the more interesting cases, such as networks with bottleneck layers (e.g. auto-encoders in Figure 1(C)).  3. For the MNIST experiment in Figure 1(C), perhaps some more comparisons are necessary. First, only comparing to SGD is not sufficient. Other first-order methods (e.g. Adams and RMSProp) or some second-order methods (e.g. Hessian-Free optimization [1]) are good potential baseline methods. Second, auto-encoding on MNIST is a well-studied task. It would be great if the authors could compare the results with the other work on this task. Third, the authors tried to compare the efficiency of their method against SGD through number of iterations, but single iteration of their method may take longer than that of SGD. It will be better if the comparison on time is also presented.  Minor comment:  Probably Section 7 is too long. I suggest maybe split it to a section of related work and a section of conclusion is better.   References:  [1] Martens J. Deep learning via Hessian-free optimization. In ICML 2010 Jun 21 (Vol. 27, pp. 735-742).  [2] Heskes, T. (2000). On natural learning and pruning in multilayered perceptrons. Neural Computation, 12(4):881–901.  [3] Povey, D., Zhang, X., and Khudanpur, S. (2014). Parallel training of dnns with natural gradient and parameter averaging. arXiv preprint arXiv:1410.7455.  [4] Desjardins, G., Simonyan, K., Pascanu, R., et al. (2015). Natural neural networks. In Advances in Neural Information Processing Systems, pages 2071–2079.  [5] Martens, J. and Grosse, R. (2015). Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, pages 2408–2417.  [6] Grosse, R. and Martens, J. (2016). A kronecker-factored approximate fisher matrix for convolution layers. In International Conference on Machine Learning, pages 573–582.