We explore some mathematical features of the loss landscape of overparameterized neural networks. A priori one might imagine that the loss function looks like a typical function from R n to R -in particular, nonconvex, with discrete global minima. In this paper, we prove that in at least one important way, the loss function of an overparameterized neural network does not look like a typical function. If a neural net has n parameters and is trained on d data points, with n > d, we show that the locus M of global minima of L is usually not discrete, but rather an n − d dimensional submanifold of R n . In practice, neural nets commonly have orders of magnitude more parameters than data points, so this observation implies that M is typically a very high-dimensional subset of R n .
This paper considers over-parametrized neural networks (n > d), where n is the number of parameters and d is the number of data, in the regression setting. It consists of three main results:
a) (Theorem 2.1) If the output of the network is a smooth function of its parameters and global minimum with zero squared loss is attainable, then the locus of global minima is a smooth $n - d$ dimensional submanifold of R^n.
b) (Lemma 3.3) A neural network with a hidden layer as wide as the number of data points can attain global minima with zero loss.
c) (Theorem 4.1) If a neural network has a hidden layer as wide as the number of data points and is a smooth function of its parameters, then the locus of global minima with loss zero is a smooth $n - d$ dimensional submanifold. This is just a combination of a) and b).
I think the only contribution of this paper is Theorem 2.1. It is already well-known (e.g., Zhang et al. 16’) that a network with a hidden layer as wide as the number of data can easily attain zero empirical error. The authors claim that it is an extension over ReLU, but the extension is almost trivial. Theorem 4.1 is just a corollary from Theorem 2.1 and Lemma 3.3.
Theorem 2.1 is an interesting observation, and its implication that the Hessian at minima has only d positive eigenvalues is interesting and explains empirical observations. 
From my understanding, however, the n - d dimensional submanifold argument can be applied to any regular values of H(w,b) (not necessarily global minima), so the theorem is basically saying there are a lot of equivalence classes in the parameter space that outputs exactly the same outcome. If my understanding is correct, this result does not give us any insight into why SGD can find global minima easily, because we can apply the same proof to spurious local minima and say, “look, the locus of this spurious local minimum is a high-dimensional submanifold, so it is easy to get stuck at poor local minima.”
Moreover, the notation used in this paper is not commonly used in deep learning community, thus might confuse the readers. $n$ is more commonly used for the number of data points and $d$ is used for some dimension. Also, there are overloading uses of $p$: 
* In Section 2.1 it is used for the dimension of the input space
* In the proof of Prop 2.3 it is now a vector in R^n
* In section 3 it is now a parameter vector of a neural network
* Back to the input dimension in Lemma 3.3 and Section 4
Also, the formulation of neural networks in Section 3 is very non-standard. If you are to end up showing some result on standard fully-connected feed-forward networks, why not just use, for example, $W_2 \sigma(W_1 x_i + b_1) + b_2$? To me, this graph theoretical formulation only adds confusion. Also at the end of page 4, there is a typo: the activation function (\sigma) must be applied after the summation.
Overall, I believe this paper does not meet the standards of ICLR at the moment. I would recommend rejection.