To provide principled ways of designing proper Deep Neural Network (DNN) models, it is essential to understand the loss surface of DNNs under realistic assumptions. We introduce interesting aspects for understanding the local minima and overall structure of the loss surface. The parameter domain of the loss surface can be decomposed into regions in which activation values (zero or one for rectified linear units) are consistent. We found that, in each region, the loss surface have properties similar to that of linear neural networks where every local minimum is a global minimum. This means that every differentiable local minimum is the global minimum of the corresponding region. We prove that for a neural network with one hidden layer using rectified linear units under realistic assumptions. There are poor regions that lead to poor local minima, and we explain why such regions exist even in the overparameterized DNNs.
This paper attempts to extend analytical results pertaining to the loss surface of linear networks to a nonlinear network with a single hidden ReLU layer.  Unfortunately though, at this point I feel that the theoretical results, which constitute the majority of the paper, are of limited novelty and/or significance.  However, I still remain very open to counterarguments to this opinion and the points raised below.
First, I don't believe that Lemma 2.2 is precisely true, at least as currently stated.  In particular, it would appear that L_f could have a differentiable local minima that is only a saddle point in L_gA.  For example, if there is a differentiable valley in L_f that terminates on the boundary of an activation region, then this phenomena could occur, since a local-minima-creating boundary in L_f might just lead to a saddle point in L_gA.  Regardless, the basic premise of this result is quite straightforward anyway.
Turning to Lemma 2.3 and 2.4, I don't understand the relevance of these results.  Where are they needed later or applied?  Additionally, Theorem 2.5 is very related to results already proven for linear networks in earlier work (Kawaguchi, 2016), so there is little novelty here.
There also seem to be issues with Corollary 2.7, which as an aggregation result can be viewed as the main contribution of the paper.  Part (1) of this corollary is obvious.  Part (2) depends on Lemma 2.2, which as stated previously may be problematic.  Most seriously though, Part (3) only considers critical points (i.e., derivative equal to zero), not local minima occurring at non-differentiable locations.  To me this greatly mutes the value of this result, and the contribution of the paper overall, because local minimum are *very* likely to occur on the boundary between activation regions at non-differentiable points (e.g. as in Figure 2).  I therefore don't understand the utility of only considering the differentiable local minima.
Overall though, the main point that within areas of fixed activation the network behaves much like a linear network (with all local minima also global minima when constrained within each region), is not especially noteworthy, because it provides no pathway for comparing minima from different activation regions, which is the main problem to begin with.
Beyond this, the paper makes a few less-technical observations regarding bad local minima.  For example, in Section 3.1 the argument is made that the linear region created when all activations are equal to one, will have a local minimum, and this minimum might be suboptimal.  However, these arguments pertain to the surrogate function L_gA, and if the minima to L_gA occurs on the boundary to another activation region, then this solution might not be a local minima to L_f, the real objective we care about.  Am I missing something here?
As for Section 4.2, the paper needs to do a better job of explaining exactly what is been shown in Table 2.  I can maybe guess, but it is not at all clear what the accuracy percentage is referring to, nor precisely how rich and random minima are computed.  Also, the assumption that P(a = 1) = P(a = 0) = 0.5 is not very realistic, although admittedly this type of simplification is sometimes adopted in the literature.
Minor comment:
* Near the beginning in the introduction, it is claimed that "the vanishing gradient problem has been solved by using rectified linear units."  This is not actually true, and portends problematic claims later in the paper.