The Spectrum of the Fisher Information Matrix of a Single-Hidden-Layer Neural Network
An important factor contributing to the success of deep learning has been the remarkable ability to optimize large neural networks using simple first-order optimization algorithms like stochastic gradient descent. While the efficiency of such methods depends crucially on the local curvature of the loss surface, very little is actually known about how this geometry depends on network architecture and hyperparameters. In this work, we extend a recently-developed framework for studying spectra of nonlinear random matrices to characterize an important measure of curvature, namely the eigenvalues of the Fisher information matrix. We focus on a single-hidden-layer neural network with Gaussian data and weights and provide an exact expression for the spectrum in the limit of infinite width. We find that linear networks suffer worse conditioning than nonlinear networks and that nonlinear networks are generically non-degenerate. We also predict and demonstrate empirically that by adjusting the nonlinearity, the spectrum can be tuned so as to improve the efficiency of first-order optimization methods.
This submission is a theoretical contribution on the spectrum of Fisher information matrix of deep neural networks. Based on random matrix theory, the authors studied such a spectrum in a very simplified setting: a one-hidden layer feed-forward network, where both the inputs and all neuron network weights are i.i.d. Gaussian distributed, all layers have the same width, the activation function has zero Gaussian mean and finite Gaussian moments, and the cost function is the squared loss. The main results are summarized in theorem 1: in the limit case (very wide layer), the Stieltjes transform of the spectral density is in closed form. The author further showed consistency of the theoretic prediction of the spectrum density with numerical simulations using several activation functions (including the linear case), which shows perfect agreement.  The theoretical results present a solid advancement on the thread of applying random matrix theory to deep learning. It is meaningful in understanding the Fisher information matrix of neural networks.  While I am convinced that this paper should be accepted by NIPS as a nice theoretical contribution, I am not familiar with random matrix theory as indicated by the confidence score.  The main comments I have is regarding the very constrained assumptions and the writing.  It is true that it is difficult to get elegant results as theorem 1 for deep learning without making simplified assumptions. However, in the end, the authors did not make any attempt on generalizing the results to more general settings, or providing relevant discussions, or making numerical simulations to show the (in)consistency of the spectrum when those assumptions fail. Therefore the results have limited values and may not have a large impact.  As a relevant question, is it possible to extend the results to the case, where dim(X) is fixed while the size of the hidden layer goes to infinity? If this case can be provided in section 3, the contribution will be much more systematical.  Did you make any experimental study on the case, where the constraints are slightly violated, while the prediction still roughly holds? If such results can be presented as figures in section 3/4, it will make the contribution more stereoscopic.  Regarding the writing, the overall technical and English quality is good. The introduction is a bit too long and deviating from the main contributions. The authors raised some big questions, such as "under which conditions first-order methods will work well". These questions are not really answered at the end. The authors are therefore suggested to streamline section 1 and "cut to" the topic. It needs less words to motivate that the spectrum of the Fisher/Hessian is important. Instead, use the saved space to provide more discussion in the end on generalizing the results. Please see the following minor comments for more details.  Minor comments:  Through out the paper there are some extra commas just before the equations. For example, L80 "can be written as , H=H^0+H^1".  Please use parentheses or brackets to denote the expectations as E_X(...). The current notation looks a bit weird.  In equation (1) there is an expectation with respect to the random variable Y. In the following equations it is gone. Please check carefully and be consistent.  Use a different symbol such as ":=" for definitions.  L101 limiting spectral density can be accompanied with an equation.  Please try to keep the paper self-contained and refer a bit less to the SM.  equation (29) can be moved to an earlier place to introduce f_opt  Figure(2,b): please make clear which curves are predictions and which curves are simulations  A relevant (unpublished) work is "Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach. Ryo Karakida, Shotaro Akaho, Shun-ichi Amari. 2018.". As this work is unpublished, it is only for the authors' interest to have a look.  -After rebuttal-  Thank you for your rebuttal. Please try include related discussions on generalizations (e.g. in the last section).