Understanding the loss surface of a neural network is fundamentally important to the understanding of deep learning. This paper presents how piecewise linear activation functions substantially shape the loss surfaces of neural networks. We first prove that the loss surfaces of many neural networks have infinite spurious local minima, which are defined as the local minima with higher empirical risks than the global minima. Our result holds for any neural network with arbitrary depth and arbitrary piecewise linear activation functions (excluding linear functions) under most loss functions in practice with some mild assumptions. This result demonstrates that the networks with piecewise linear activations possess substantial differences to the well-studied linear neural networks. Essentially, the underlying assumptions for the above result are consistent with most practical circumstances where the output layer is narrower than any hidden layer. In addition, the loss surface of a neural network with piecewise linear activations is partitioned into multiple smooth and multilinear open cells by nondifferentiable boundaries. The constructed spurious local minima are exactly connected with each other in one cell by a continuous path, on which the empirical risk is invariant. We further prove that within every cell of a one-hidden-layer network, local minima are equally good, and also, they are all global minima in the cell.
Summary: This paper studies the landscape of deep neural networks with piecewise-linear activation functions. The paper showed that under very mild assumptions, the loss surface admits infinite spurious local minima. Further, it is shown that the loss surface is partitioned into many multilinear cells. If the network is two-layer with two-piece linear activations, it is proved that within each cell every local minimum is global. 
Pros:
  --Constructed spurious local minima for piecewise linear activations, for a broader setting than previous papers. 
  --The paper is well written, with detailed explanation of proof skeleton. 
Cons: 
The significance of the results are not clear. Details are given below. 
1.	This paper only considers piecewise linear activation, which is a special type of non-linear activation. The major examples are ReLU and leaky ReLU. However, related results for ReLU have been studied for a few previous works mentioned in the last two paragraphs of Sec. 3.2. In particular, Yun et al. 2019b already proves a similar result for 1-hidden-layer neural-net with ReLU activation. I think extending the construction to broader settings (any depth, any piecewise linear and more losses) is mathematically nice, but the motivation of this extension is somewhat unclear to me. One motivation is that this is helpful for the purpose of understanding a “big picture” of the landscape, which I will discuss next.
2.	The second major result is Theorem 2, on the “big picture” with ReLU-like activations. However, Theorem 2 is somewhat trivial to prove, and the link to Theorem 1 is rather weak.
   (a) The main message of Thm 2 is the partition of the surface into multiple pieces, and each piece has good property. This partition is somewhat straightforward, and has been studied before, in, e.g., [R1].
   For a global “big picture”, partitioning itself is not very interesting. Theorem 2 mainly describes the property of each region separately for 2-layer network, which is weaker than [R1]. 
   (b) Theorem 2 seems easy to prove. The 1st, 3rd and 4th property are all straightforward. The 2nd property “local analogous convexity” was given a 2-page proof in the paper. However, I don’t understand why not use the following simple argument: for each region, the network behaves like a deep linear network, thus directly applying existing result shall imply “every local minimum in the region is the global minimum of the region”, right? If not, what is the difficulty?
   (c) The 3rd property says “some local minima are concentrated as a valley in some cell”. What are the formal definitions of “concentrated” and “valley” in this sentence?
   (d) The link to Theorem 1 is weak: the link is the 3rd property of Theorem 2 that “some local minima are in a valley”. It is just about some special local minima and weakly related to the other properties on the “global view”. In addition, the fact that “some of them are in a valley” may be due to the very special construction, thus it is not surprising and does not reveal anything interesting about the “big picture”. 
3.	Other issues:
a) While ReLU-type activations are popular, there are still commonly-used activation functions are not piece-wise linear, e.g., tanh, swish. It is not proper to claim that "this paper presents how nonlinearities in activations substantially shape the loss surface" and "almost every practical neural network ....". I suggest replacing "nonlinearity" with "piecewise linearity" in both the title and the abstract, and modifying the over-statements. 
   b) In Property 1 of Theorem 2, “smooth and multilinear partition” might be a bit misleading. The loss surface should be fractional in general, where multilinear cells are separated by non-smooth boundaries. “Smooth partition” seems to imply that the boundaries are smooth or the partition method is smooth in some sense.
   c) The name “analogous convexity” is not appropriate. Analogous convexity is not formally defined in the paper. According to Sec. 4.3 third paragraph, “the property of analogous convexity that the local minima wherein are equally good”. It seems that “analogous convexity” is just “all local minima are good”, which is very different from convexity. It is a weaker property than quasi-convexity, star-convexity, etc, and thus it is better not to call it “analogous convexity”.
    d) Property 3 of Theorem 2 is very far from “mode connectivity”. The proof of Property 3 relies on a special construction of Theorem 1, and the latter is for two arbitrary global minima. 
[R1] Soudry and Hoffer. "Exponentially vanishing sub-optimal local minima in multilayer neural networks." arXiv preprint arXiv:1702.05777 (2017).
Conclusion:  I think this paper is studying an important and interesting question, and the efforts of constructing local minima and understanding big picture are both interesting to me. However, I’m afraid the current form of the paper does not meet the standard of the conference. That being said, it would be a nice paper if the big picture can be explored deeper, and the link to the spurious local minima can be built stronger.