We study the properties of common loss surfaces through their Hessian matrix. In particular, in the context of deep learning, we empirically show that the spectrum of the Hessian is composed of two parts: (1) the bulk centered near zero, (2) and outliers away from the bulk. We present numerical evidence and mathematical justifications to the following conjectures laid out by : Fixing data, increasing the number of parameters merely scales the bulk of the spectrum; fixing the dimension and changing the data (for instance adding more clusters or making the data less separable) only affects the outliers. We believe that our observations have striking implications for non-convex optimization in high dimensions. First, the flatness of such landscapes (which can be measured by the singularity of the Hessian) implies that classical notions of basins of attraction may be quite misleading. And that the discussion of wide/narrow basins may be in need of a new perspective around over-parametrization and redundancy that are able to create large connected components at the bottom of the landscape. Second, the dependence of a small number of large eigenvalues to the data distribution can be linked to the spectrum of the covariance matrix of gradients of model outputs. With this in mind, we may reevaluate the connections within the data-architecturealgorithm framework of a model, hoping that it would shed light on the geometry of high-dimensional and non-convex spaces in modern applications. In particular, we present a case that links the two observations: small and large batch gradient descent appear to converge to different basins of attraction but we show that they are in fact connected through their flat region and so belong to the same basin.
This paper has at its core an interesting, novel, tentative claim, backed up by simple experiments, that small batch gradient descent and large batch gradient descent may converge to points in the same basin of attraction, contrary to the discussion (but not the actual experimental results) of Keskar et al. (2016). In general, there is a pressing need for insight into the qualitative behavior of gradient-based optimization and this area is of immense interest to many machine learning practitioners. Unfortunately the interesting tentative insights are surrounded by many unsubstantiated and only tangentially related theoretical discussions. Overall the paper has the appearance of lacking a sharp focus. This is a shame since I found the core of the paper very interesting and thought provoking.
Major comments:
While the paper has some interesting tentative experimental insights, the relationship between theory and experiment is complicated. The theoretical claims are vague and wide ranging, and are not all individually well supported or even tested by the experiments. Rather than including lots of small potential insights which the authors have had about what may be going on during gradient-based optimization, I'd prefer to see a paper with much tighter focus with a small number of theoretical claims well supported by experiments (it's fine if the experiments are simplistic as here; that's still interesting).
A large amount of the paper hinges on being able to ignore the second term in (6), and this fact is referred to many times, but the theoretical and experimental justification for this claim is very thin.
The authors mention overparameterization repeatedly, and it's in the title, but they never define it. It also doesn't appear to take center stage in their experimental investigations (if it is in fact critical to the experiments then it should be made clearer how).
Throughout this paper there is not a clear distinction between eigenvalues being zero and eigenvalues being close to zero, or similarly between the Hessian being singular and ill-conditioned. This distinction is particularly important in the theoretical discussion.
It would be helpful to be clearer about the differences between this work and that presented in Sagun et al. (2016).
Minor comments:
The assumption that the target y is real is at odds with many regression problems and practically all classification. It might be worth generalizing the discussion to multidimensional targets.
It would be good to have some citations to support the claim that often "the number of parameters M is comparable to the number of examples N (if not much larger)". With 1-dimensional targets as considered here, that sounds like a recipe for extreme overfitting and poor generalization. Generically based on counting constraints and free parameters one would expect to be able to fit exactly any dataset of N output values using a model with M free parameters. (With P-dimensional targets the relevant comparison would be M vs N P rather than M vs N).
At the end of intro to section 1, "loss is non-degenerate" should be "Hessian of the loss is non-degenerate"? Also, didn't the paper cited assume at least one negative eigenvalue at any saddle point, rather than non-degeneracy?
In section 1.1, it would be helpful to explain the precise sense in which "overparameterized" is being used. Hopefully it is in the sense that there are more parameters than needed for good performance at the true global minimum (the additional parameters helping with the process of *finding* a good minimum rather than its existence) or in the sense that M -> infinity for N "equal to" infinity. If it is in the sense that M >> N then I'm not sure of the relevance to practical machine learning.
It would be helpful to use a log scale for the plot in Figure 1. The claim that the Hessian is ill-conditioned depends on the condition number, which is impossible to estimate from the plot.
The fact that "wide basins, as opposed to narrow ones, generalize better" is not a new claim of the Keskar et al. paper. I'd argue it's well-known and part of the classical explanation of why maximum likelihood methods overfit and Bayesian ones don't. See for example MacKay, Information Theory Inference and Learning Algorithms.
"It turns out that the Hessian is degenerate at any given point" makes it sound like the result is a theoretical one. As I understand it, the experimental investigation in Sagun et al. (2016) just shows that the Hessian may often be ill-conditioned. As above, more clarity is also needed about whether it is literally degenerate or just approximately so, in which case ill-conditioned is probably a more appropriate word. Ill-conditioned is also more appropriate than singular in "slightly singular but extremely so".
How much data was used for the simple experiments in Figure 1? Infinite data? What data was used?
It would be helpful to spell out the intuition in "Intuitively, this kind of singularity...".
I don't think the decomposition (5) is required to "explain why having more parameters than samples results in degenerate Hessian matrices". Generically one would expect that with 1-dimensional targets, N datapoints and N + Q parameters, there would be a Q-dimensional submanifold of parameter space on which the loss would be zero. Of course there would be a few conditions needed to make this into a precise statement, but no need for assuming the second term is negligible.
Is the conventional decomposition of the loss into l o f used for the generalized Gauss Newton that f is a function only of the input to the neural net and the model parameters, but not the target? I could be wrong, but that was always my interpretation.
It's not clear whether the phrase "bottom of the landscape" used several times in the paper refers to the neighborhood of local minima or of global minima.
What is the justification for assuming l'(f(w)) and grad f(w) are not correlated? That seems unlikely to be true in general! Also spell out why this implies the second term can be ignored. I'm a bit skeptical of the claim in general. It's easy to come up with counterexamples. For example take l to be the identity (say f has a relu applied to it to ensure everything is well formed).
"Immediately, this implies that there are at least M - N trivial eigenvalues of the Hessian". Make it clear that trivial here means approximately not exactly zero (in which case a good word would be "small"); this follows since the second term in (5) is only approximately zero. In fact it should be possible to prove there are M - N values which are exactly zero, but that doesn't follow from the argument presented. As above I'd argue this analysis is somewhat beside the point since N should be greater than M in practice to prevent severe overfitting.
In section 3.1, "trivial eigenvalues" should be "non-trivial eigenvalues".
What's the relevance of using PCA on the data in Figure 2 when it comes to analyzing training neural nets? Also, is there any reason 2 classes breaks the trend?
What size of data was used for the experiments to plot figure 2 and figure 3? Infinite?
It's not completely clear what the takeaway is from Figure 3. I presume this is supporting the point that the eigenvalues of the Hessian at convergence consist of a bulk and outliers. The could be stated explicitly. Is there any significance to the fact that the number of clusters is equal to the number of outliers? Is this supporting some broader claim of the paper?
Figure 4, 5, 6 would benefit from being log plots, and make the claim that the bulk has the same shape independent of data much stronger.
The x-axis in Figure 5 is not "ordered counts of eigenvalues" but "index of eigenvalues", and in Figure 6 is not "ratios of eigenvalues" but ratio of the index. In the caption for Figure 6, "scaled by their ratio" is not clear.
I don't follow why Figure 6 confirms that "the effects of the ignored term in the decomposition is small" for negative eigenvalues.
In section 3.3, when saying the variances of the steps are different but the means are similar, it may interesting to note that the variance is often the dominant term and much greater in magnitude than the mean when doing SGD (at least that's what I've experienced).
What's the meaning of "elbow at similar levels"? What's the significance?
In section 4 it is claimed that overparameterization is what "leads to flatness at the bottom of the landscape which is easy to optimize". The bulk-outlier view suggests that adding extra parameters may just add extra dimensions to the flat region, but why is optimizing 100 values in a flat 100-dimensional space easier than optimizing 10 values in a flat 10-dimensional space?
In section 4.1, "fair comparison" is misleading since it depends on perspective. If one cares about compute time then certainly measuring steps rather than epochs would not be a fair comparison!
What's the relevance of the fact that random initial points in high-dimensional spaces are almost always nearly orthogonal (N.B. the "nearly" should be added)? This seems to be assuming something about the mapping from initial point to basin of attraction.
What's the meaning of "extending away from either end points appear to be confirming the sharpness of [the] LB solution"? Is this shown somewhere?
It would be helpful to highlight the key difference to Keskar et al. (which I believe is initializing SB training from LB point rather than from scratch). I presume the claim is that Keskar et al. only found their "inverted camel hump" linear interpolation results due to the random initialization, and that this would also often be observed for, say, two random LB-from-scratch trainings (which may randomly fall into different basins of attraction). If this is the intended point then it would be good to make this explicit.
In "the first terms starts to dominate", to dominate what? The gradient, or the second term in (5)? If the latter, what is the relevance of this?
Why "even" in "Even when the weight space has large flat regions"?
In the last paragraph of section 4.1, it might be worth spelling out that (as I understand it) the idea is that the small batch method finds itself in a poor region to begin with, since the average loss over an SB-noise-sized neighborhood of the LB point is actually not very good, and so there is a non-zero gradient through flat space to a place where the average loss over an SB-noise-sized neighborhood is good.
In section 5, "we see that even large batch methods are able to get to the level where small batch methods go" seems strange. Isn't this of training set loss? Isn't the "level" people care about the test set loss?
In appendix A, the meaning of consecutive in "largest consecutive gap" and "largest consecutive ratio" was not clear to me.
Appendix B is only referred to in a footnote. What is its significance for the main theme of the paper? I'd suggest either making it more prominent or putting it in a separate paper.