Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
We study the properties of common loss surfaces through their Hessian matrix. In particular, in the context of deep learning, we empirically show that the spectrum of the Hessian is composed of two parts: (1) the bulk centered near zero, (2) and outliers away from the bulk. We present numerical evidence and mathematical justifications to the following conjectures laid out by Sagun et. al. (2016): Fixing data, increasing the number of parameters merely scales the bulk of the spectrum; fixing the dimension and changing the data (for instance adding more clusters or making the data less separable) only affects the outliers. We believe that our observations have striking implications for non-convex optimization in high dimensions. First, the *flatness* of such landscapes (which can be measured by the singularity of the Hessian) implies that classical notions of basins of attraction may be quite misleading. And that the discussion of wide/narrow basins may be in need of a new perspective around over-parametrization and redundancy that are able to create *large* connected components at the bottom of the landscape. Second, the dependence of a small number of large eigenvalues to the data distribution can be linked to the spectrum of the covariance matrix of gradients of model outputs. With this in mind, we may reevaluate the connections within the data-architecture-algorithm framework of a model, hoping that it would shed light on the geometry of high-dimensional and non-convex spaces in modern applications. In particular, we present a case that links the two observations: small and large batch gradient descent appear to converge to different basins of attraction but we show that they are in fact connected through their flat region and so belong to the same basin.
 The authors perform a set of experiments in which they inspect the Hessian matrix of the loss of a neural network, and observe that most of the eigenvalues are very close to zero. This is a potentially important observation, and the experiments were well worth performing, but I don*t find them fully convincing (partly because I was confused by the presentation). They perform four sets of experiments: 1) In section 3.1, they show on simulated data that for data drawn from k clusters, there are roughly k significant eigenvalues in the Hessian of the solution. 2) In section 3.2, they show on MNIST that the solution contains few large eigenvalues, and also that there are negative eigenvalues. 3) In section 3.3, they show (again on MNIST) that at their respective solutions, large batch and small batch methods find solutions with similar numbers of large eigenvalues, but that for the large batch method the magnitudes are larger. 4) In section 4.1, they train (on CIFAR10) using a large batch method, and then transition to a small batch method, and argue that the second solution appears to be better than the first, but that they are a part of the same basin (since linearly while interpolating between them they don*t run into any barriers). I*m not fully convinced by the second and third experiments, partly because I didn*t fully understand the plots (more on this below), but also because it isn*t clear to me what we should expect from the spectrum of a Hessian, so I don*t know whether the observed specra have fewer large eigenvalues, or more large eigenvalues, then would be "natural". In other words, there isn*t a *baseline*. For the fourth experiment, it*s unsurprising that the small batch method winds up in a different location in the same basin as the large batch method, since it was initialized to the large batch method*s solution (and it doesn*t appear to me, in figure 9, that the small batch solution is significantly different). Section 2.1 is said to contain an argument that the second term of equation 5 can be ignored, but only says that if .ell* and .nabla^2 of f are uncorrelated, then it can be ignored. I don*t see any reason that these two quantities should be correlated, but this is not an argument that they are uncorrelated. Also, it isn*t clear to me where this approximation was used--everywhere? In section 3.2, it sounds as if the exact Hessian is used, and at the end of this section the authors say that figure 6 demonstrates that the effect of this second term is small, but I don*t see why this is, and it isn*t explained. My main complaint is that I had a great deal of difficulty interpreting the plots: it often wasn*t clear to me what exactly was being plotted, and most of the language describing them was frustratingly vague. For example, figure 6 is captioned "left edge of the spectrum, eigenvalues are scaled by their ratio". The text explains that "left edge of the spectrum" means "small but negative eigenvalues" (this would be better in the caption), but what are the ratios? Ratio of what to what? I think it would greatly enhance clarity if every plot caption described exactly, and unambiguously, what quantities were plotted on the horizontal and vertical axes. Some minor notes: There are a number of places where "it*s" is used, where it should be "its". In the introduction, the definition of .mathcal{L}* is slightly confusing, since it*s an expectation, but the use of "*" makes one expect a derivative. Perhaps use .hat{.mathcal{L}} for the empirical loss, and .mathcal{L} for the expected one? On the bottom of page 4, "if .ell* and .nabla f are not correlated": I think the .nabla should be .nabla^2. It*s "principal components", not "principle components".
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
We study the properties of common loss surfaces through their Hessian matrix. In particular, in the context of deep learning, we empirically show that the spectrum of the Hessian is composed of two parts: (1) the bulk centered near zero, (2) and outliers away from the bulk. We present numerical evidence and mathematical justifications to the following conjectures laid out by Sagun et. al. (2016): Fixing data, increasing the number of parameters merely scales the bulk of the spectrum; fixing the dimension and changing the data (for instance adding more clusters or making the data less separable) only affects the outliers. We believe that our observations have striking implications for non-convex optimization in high dimensions. First, the *flatness* of such landscapes (which can be measured by the singularity of the Hessian) implies that classical notions of basins of attraction may be quite misleading. And that the discussion of wide/narrow basins may be in need of a new perspective around over-parametrization and redundancy that are able to create *large* connected components at the bottom of the landscape. Second, the dependence of a small number of large eigenvalues to the data distribution can be linked to the spectrum of the covariance matrix of gradients of model outputs. With this in mind, we may reevaluate the connections within the data-architecture-algorithm framework of a model, hoping that it would shed light on the geometry of high-dimensional and non-convex spaces in modern applications. In particular, we present a case that links the two observations: small and large batch gradient descent appear to converge to different basins of attraction but we show that they are in fact connected through their flat region and so belong to the same basin.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The authors perform a set of experiments in which they inspect the Hessian matrix of the loss of a neural network, and observe that most of the eigenvalues are very close to zero. This is a potentially important observation, and the experiments were well worth performing, but I don*t find them fully convincing (partly because I was confused by the presentation). They perform four sets of experiments: 1) In section 3.1, they show on simulated data that for data drawn from k clusters, there are roughly k significant eigenvalues in the Hessian of the solution. 2) In section 3.2, they show on MNIST that the solution contains few large eigenvalues, and also that there are negative eigenvalues. 3) In section 3.3, they show (again on MNIST) that at their respective solutions, large batch and small batch methods find solutions with similar numbers of large eigenvalues, but that for the large batch method the magnitudes are larger. 4) In section 4.1, they train (on CIFAR10) using a large batch method, and then transition to a small batch method, and argue that the second solution appears to be better than the first, but that they are a part of the same basin (since linearly while interpolating between them they don*t run into any barriers). I*m not fully convinced by the second and third experiments, partly because I didn*t fully understand the plots (more on this below), but also because it isn*t clear to me what we should expect from the spectrum of a Hessian, so I don*t know whether the observed specra have fewer large eigenvalues, or more large eigenvalues, then would be "natural". In other words, there isn*t a *baseline*. For the fourth experiment, it*s unsurprising that the small batch method winds up in a different location in the same basin as the large batch method, since it was initialized to the large batch method*s solution (and it doesn*t appear to me, in figure 9, that the small batch solution is significantly different). Section 2.1 is said to contain an argument that the second term of equation 5 can be ignored, but only says that if .ell* and .nabla^2 of f are uncorrelated, then it can be ignored. I don*t see any reason that these two quantities should be correlated, but this is not an argument that they are uncorrelated. Also, it isn*t clear to me where this approximation was used--everywhere? In section 3.2, it sounds as if the exact Hessian is used, and at the end of this section the authors say that figure 6 demonstrates that the effect of this second term is small, but I don*t see why this is, and it isn*t explained. My main complaint is that I had a great deal of difficulty interpreting the plots: it often wasn*t clear to me what exactly was being plotted, and most of the language describing them was frustratingly vague. For example, figure 6 is captioned "left edge of the spectrum, eigenvalues are scaled by their ratio". The text explains that "left edge of the spectrum" means "small but negative eigenvalues" (this would be better in the caption), but what are the ratios? Ratio of what to what? I think it would greatly enhance clarity if every plot caption described exactly, and unambiguously, what quantities were plotted on the horizontal and vertical axes. Some minor notes: There are a number of places where "it*s" is used, where it should be "its". In the introduction, the definition of .mathcal{L}* is slightly confusing, since it*s an expectation, but the use of "*" makes one expect a derivative. Perhaps use .hat{.mathcal{L}} for the empirical loss, and .mathcal{L} for the expected one? On the bottom of page 4, "if .ell* and .nabla f are not correlated": I think the .nabla should be .nabla^2. It*s "principal components", not "principle components".
assistant
 The authors perform a set of experiments in which they inspect the Hessian matrix of the loss of a neural network, and observe that most of the eigenvalues are very close to zero. This is a potentially important observation, and the experiments were well worth performing, but I don*t find them fully convincing (partly because I was confused by the presentation). They perform four sets of experiments: 1) In section 3.1, they show on simulated data that for data drawn from k clusters, there are roughly k significant eigenvalues in the Hessian of the solution. 2) In section 3.2, they show on MNIST that the solution contains few large eigenvalues, and also that there are negative eigenvalues. 3) In section 3.3, they show (again on MNIST) that at their respective solutions, large batch and small batch methods find solutions with similar numbers of large eigenvalues, but that for the large batch method the magnitudes are larger. 4) In section 4.1, they train (on CIFAR10) using a large batch method, and then transition to a small batch method, and argue that the second solution appears to be better than the first, but that they are a part of the same basin (since linearly while interpolating between them they don*t run into any barriers). I*m not fully convinced by the second and third experiments, partly because I didn*t fully understand the plots (more on this below), but also because it isn*t clear to me what we should expect from the spectrum of a Hessian, so I don*t know whether the observed specra have fewer large eigenvalues, or more large eigenvalues, then would be "natural". In other words, there isn*t a *baseline*. For the fourth experiment, it*s unsurprising that the small batch method winds up in a different location in the same basin as the large batch method, since it was initialized to the large batch method*s solution (and it doesn*t appear to me, in figure 9, that the small batch solution is significantly different). Section 2.1 is said to contain an argument that the second term of equation 5 can be ignored, but only says that if .ell* and .nabla^2 of f are uncorrelated, then it can be ignored. I don*t see any reason that these two quantities should be correlated, but this is not an argument that they are uncorrelated. Also, it isn*t clear to me where this approximation was used--everywhere? In section 3.2, it sounds as if the exact Hessian is used, and at the end of this section the authors say that figure 6 demonstrates that the effect of this second term is small, but I don*t see why this is, and it isn*t explained. My main complaint is that I had a great deal of difficulty interpreting the plots: it often wasn*t clear to me what exactly was being plotted, and most of the language describing them was frustratingly vague. For example, figure 6 is captioned "left edge of the spectrum, eigenvalues are scaled by their ratio". The text explains that "left edge of the spectrum" means "small but negative eigenvalues" (this would be better in the caption), but what are the ratios? Ratio of what to what? I think it would greatly enhance clarity if every plot caption described exactly, and unambiguously, what quantities were plotted on the horizontal and vertical axes. Some minor notes: There are a number of places where "it*s" is used, where it should be "its". In the introduction, the definition of .mathcal{L}* is slightly confusing, since it*s an expectation, but the use of "*" makes one expect a derivative. Perhaps use .hat{.mathcal{L}} for the empirical loss, and .mathcal{L} for the expected one? On the bottom of page 4, "if .ell* and .nabla f are not correlated": I think the .nabla should be .nabla^2. It*s "principal components", not "principle components".