The loss landscape of overparameterized neural networks
We explore some mathematical features of the loss landscape of overparameterized neural networks. A priori one might imagine that the loss function looks like a typical function from to - in particular, nonconvex, with discrete global minima. In this paper, we prove that in at least one important way, the loss function of an overparameterized neural network does not look like a typical function. If a neural net has parameters and is trained on data points, with , we show that the locus of global minima of is usually not discrete, but rather an dimensional submanifold of . In practice, neural nets commonly have orders of magnitude more parameters than data points, so this observation implies that is typically a very high-dimensional subset of .
 This paper gave an interesting theoretical result that the global minima of an overparameterized neural network is a high-dimensional sub-manifold. This result is particularly meaningful as it connected several previous observations about neural networks and a indirect evidence for why overparameterization for deep learning has been so helpful empirically. The proof in the paper was smart and the rough logic was quite easy to follow. The minor issue is that the proof in the paper was too sketchy to be strict. For example, in proof for Thm 2.1, the original statement about Sard’s theorem was about the critical values, but the usage of this theorem in the proof was a little indirect. I can roughly see how the logic can go through, but I still hope the author can give more detailed explaining about this part to make the proof more readable and strict. Overall, I think the result in this paper should be enough to justify a publication. However, there’re still limitations of the result here. For example, the result only explained about the fitting on training data but cannot explain at all why overfitting is not a concern here. It also didn’t explain why stochastic gradient descent can find these minima empirically. In particular, even though the minima manifold is n-d dimensional, it’s still a zero-measure set which will almost never get hit with a random initialization. Of course, these are harder questions to explore, but maybe worthy some discussion in the final revision.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
The loss landscape of overparameterized neural networks
We explore some mathematical features of the loss landscape of overparameterized neural networks. A priori one might imagine that the loss function looks like a typical function from to - in particular, nonconvex, with discrete global minima. In this paper, we prove that in at least one important way, the loss function of an overparameterized neural network does not look like a typical function. If a neural net has parameters and is trained on data points, with , we show that the locus of global minima of is usually not discrete, but rather an dimensional submanifold of . In practice, neural nets commonly have orders of magnitude more parameters than data points, so this observation implies that is typically a very high-dimensional subset of .
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper gave an interesting theoretical result that the global minima of an overparameterized neural network is a high-dimensional sub-manifold. This result is particularly meaningful as it connected several previous observations about neural networks and a indirect evidence for why overparameterization for deep learning has been so helpful empirically. The proof in the paper was smart and the rough logic was quite easy to follow. The minor issue is that the proof in the paper was too sketchy to be strict. For example, in proof for Thm 2.1, the original statement about Sard’s theorem was about the critical values, but the usage of this theorem in the proof was a little indirect. I can roughly see how the logic can go through, but I still hope the author can give more detailed explaining about this part to make the proof more readable and strict. Overall, I think the result in this paper should be enough to justify a publication. However, there’re still limitations of the result here. For example, the result only explained about the fitting on training data but cannot explain at all why overfitting is not a concern here. It also didn’t explain why stochastic gradient descent can find these minima empirically. In particular, even though the minima manifold is n-d dimensional, it’s still a zero-measure set which will almost never get hit with a random initialization. Of course, these are harder questions to explore, but maybe worthy some discussion in the final revision.
assistant
 This paper gave an interesting theoretical result that the global minima of an overparameterized neural network is a high-dimensional sub-manifold. This result is particularly meaningful as it connected several previous observations about neural networks and a indirect evidence for why overparameterization for deep learning has been so helpful empirically. The proof in the paper was smart and the rough logic was quite easy to follow. The minor issue is that the proof in the paper was too sketchy to be strict. For example, in proof for Thm 2.1, the original statement about Sard’s theorem was about the critical values, but the usage of this theorem in the proof was a little indirect. I can roughly see how the logic can go through, but I still hope the author can give more detailed explaining about this part to make the proof more readable and strict. Overall, I think the result in this paper should be enough to justify a publication. However, there’re still limitations of the result here. For example, the result only explained about the fitting on training data but cannot explain at all why overfitting is not a concern here. It also didn’t explain why stochastic gradient descent can find these minima empirically. In particular, even though the minima manifold is n-d dimensional, it’s still a zero-measure set which will almost never get hit with a random initialization. Of course, these are harder questions to explore, but maybe worthy some discussion in the final revision.