Background: Statistical mechanics results (Dauphin et al. (2014); Choromanska et al. (2015)) suggest that local minima with high error are exponentially rare in high dimensions. However, to prove low error guarantees for Multilayer Neural Networks (MNNs), previous works so far required either a heavily modified MNN model or training method, strong assumptions on the labels (e.g., "near" linear separability), or an unrealistically wide hidden layer with Ω (N ) units. Results: We examine a MNN with one hidden layer of piecewise linear units, a single output, and a quadratic loss. We prove that, with high probability in the limit of N → ∞ datapoints, the volume of differentiable regions of the empiric loss containing sub-optimal differentiable local minima is exponentially vanishing in comparison with the same volume of global minima, given standard normal input of dimension d 0 =Ω √ N , and a more realistic number of
hidden units. We demonstrate our results numerically: for example, 0% binary classification training error on CIFAR with only N/d 0 ≈ 16 hidden neurons.
## Summary
This paper aims to tackle the question: "why does standard SGD based algorithms on neural network converge to 'good' solutions?" 
Pros: 
Authors ask the question of convergence of optimization (ignoring generalization error): how "likely" is that an over-parameterized (d1d0 > N) single hidden layer binary classifier "find" a good (possibly over-fitted) local minimum. They make a set of assumptions (A1-A3) which are weaker (d1 > N^{1/2}) than the ones used earlier works. Previous works needed a wide hidden layer (d1 > N).
Assumptions (d0=input dim, d1=hidden dim, N=n of datapoints, X=datapoints matrix):
A1. Datapoints X come from a Gaussian distribution 
A2. N^{1/2} < d0 =< N
A3. N polylog(N) < d0d1 (approximate n of. parameters)  and d1 =< N
This paper proves that total "angular volume" of "regions" (defined with respect to the piecewise linear regions of neuron activations) with differentiable bad-local minima are exponentially small when compared with to the total "angular volume" of "regions" containing only differentiable global-minimal. The proof boils down to counting arguments and concentration inequality.
Cons: 
Non-differentiable stationary points are left as a challenging future work on this paper. Non-differentiability aside, authors show a possible way by which shallow neural networks might be over-fitting the data. But this is only half the story and does not completely answer the question. First, exponentially vanishing (in N) volume of the "regions" containing bad-local minima doesn't mean that the number of bad local minima are exponentially small when compared to number global minima. 
Secondly, as the authors aptly pointed out in the discussion section, this results doesn't mean neural networks will converge to good local minima because these bad local minimas can have a large basins of attraction.
Lastly, appropriate comparisons with the existing literature is lacking. It is hinted that this paper is more general as the assumptions are more realistic. However, it comes at a cost of losing sharpness in the theoretical results. It is not well motivated why one should study the angular volume of the global and local minima. 
## Questions and comments
1. How critical is Gaussian-datapoints assumption (A1)? Which part of the proof fails to generalize?  
2. Can the proof be extended to scalar regression?  It seems hard to generalize to vector output neural networks. What about deep neural networks? 
3. Can you relate the results to other more recent works like: https://arxiv.org/pdf/1707.04926.pdf.
4. Piecewise linear and positively homogeneous (https://arxiv.org/pdf/1506.07540.pdf) activation seem to be important assumption of the paper. It should probably be mentioned explicitly.
5. In the experiments section, it is mentioned that "...inputs to the hidden neurons converge to a distinctly non-zero value. This indicates we converged to DLMs." How can you guarantee that it is a local minimum and not a saddle point?