In this paper, we study the problem of optimizing a two-layer artificial neural network that best fits a training dataset. We look at this problem in the setting where the number of parameters is greater than the number of sampled points. We show that for a wide class of differentiable activation functions (this class involves most nonlinear functions and excludes piecewise linear functions), we have that arbitrary first-order optimal solutions satisfy global optimality provided the hidden layer is non-singular. We essentially show that these non-singular hidden layer matrix satisfy a "good" property for these big class of activation functions. Techniques involved in proving this result inspire us to look at a new algorithmic, where in between two gradient step of hidden layer, we add a stochastic gradient descent (SGD) step of the output layer. In this new algorithmic framework, we extend our earlier result and show that for all finite iterations the hidden layer satisfies the"good" property mentioned earlier therefore partially explaining success of noisy gradient methods and addressing the issue of data independency of our earlier result. Both of these results are easily extended to hidden layers given by a flat matrix from that of a square matrix. Results are applicable even if network has more than one hidden layer provided all inner hidden layers are arbitrary, satisfy non-singularity, all activations are from the given class of differentiable functions and optimization is only with respect to the outermost hidden layer. Separately, we also study the smoothness properties of the objective function and show that it is actually Lipschitz smooth, i.e., its gradients do not change sharply. We use smoothness properties to guarantee asymptotic convergence of O(1/number of iterations) to a first-order optimal solution.
This paper aims to study some of the theoretical properties of the global optima of single-hidden layer neural networks and also the convergence to such a solution. I think there are some interesting arguments made in the paper e.g. Lemmas 4.1, 5.1, 5.2, and 5.3. However, as I started reading beyond intro I increasingly got the sense that this paper is somewhat incomplete e.g. while certain claims are made (abstract/intro) the theoretical justification are rather far from these claims. Of course there is a chance that I might be misunderstanding some things and happy to adjust my score based on the discussions here.
Detailed comments:
1) My main concern is that the abstract and intro claims things that are never proven (or even stated) in the rest of the paper
Example 1 from abstract: 
“We show that for a wide class of differentiable activation functions (this class involved “almost” all functions which are not piecewise linear), we have that first-order optimal solutions satisfy global optimality provided the hidden layer is non-singular.”
This is certainly not proven and in fact not formally stated anywhere in the paper. Closest result to this is Lemma 4.1 however, because the optimal solution is data dependent this lemma can not be used to conclude this. 
Example 2 from intro when comparing with other results on page 2:
The authors essentially state that they have less restrictive assumptions in the form of the network or assumptions on the data (e.g. do not require Gaussianity). However as explained above the final conclusions are also significantly weaker than this prior literature so it’s a bit of apples vs oranges comparison.
2) Page 2 minor typos
We study training problem -->we study the training problem
In the regime training objective--> in the regime the training objective
3) the basic idea argument and derivative calculations in section 3 is identical to section 4 of Soltan...et al
4) Lemma 4.1 is nice, well done! That being said it does not seem easy to make it (1) quantifiable (2) apply to all W. It would also be nice to compare with Soudry et. al.
5) Argument on top of page 6 is incorrect as the global optima is data dependent and hence lemma 4.1 (which is for a fixed matrix) does not apply
6) Section 5 on page 6. Again the stated conclusion here that the iterates do not lead to singular W is much weaker than the claims made early on.
 
7) I haven’t had time yet to verify correctness of Lemmas 5.1, 5.2, and Lemma 5.3 in detail but if this holds is a neat argument to side step invertibility w.r.t. W, Nicely done!
8) What is the difference between Lemma 5.4 and Lemma 6.12 of Soltan...et al 
9) Theorem 5.9. Given that the arguments in this paper do not show asymptotic convergence to a point where gradient vanishes and W is invertible why is the proposed algorithm better than a simple approach in which gradient descent is applied but a small amount of independent Gaussian noise is injected in every iteration over W. By adjusting the noise variance across time one can ensure a result of the kind in Theorem 5.9 (Of course in the absence of a quantifiable version of Lemma 4.1 which can apply to all W that result will also suffer from the same issues).