We show that for any convex differentiable loss, a deep linear network has no spurious local minima as long as it is true for the two layer case. This reduction greatly simplifies the study on the existence of spurious local minima in deep linear networks. When applied to the quadratic loss, our result immediately implies the powerful result by Kawaguchi (2016). Further, with the recent work by Zhou & Liang (2018), we can remove all the assumptions in (Kawaguchi, 2016). This property holds for more general "multi-tower" linear networks too. Our proof builds on the work in (Laurent & von Brecht, 2018) and develops a new perturbation argument to show that any spurious local minimum must have full rank, a structural property which can be useful more generally.
The motivation of this paper is training deep neural network seems to not suffer from local minima, and it tries to explain this phenomenon by showing that all local minima of deep neural network is global minima. The paper shows that for any convex differentiable loss function, a deep linear neural network has no so called spurious local minima, which to be specific, are local minima that are not global minima, as long as it is true for two-layer Neural Network. The motivation is that combining with existing result that no spurious local minima exists for quadratic loss in two-layer Neural Network, this relation connecting between two-layer and deeper linear neural network immediately implies an existing result that all local minima are global minima, removing all assumptions. The result also holds for general “multi-tower” linear networks. 
Overall, this paper could be an improvement of existing results. It is well written and the proof step is clear in general. However, there’re some weakness need clarifications on the results, especially on the novelty. Given reasonable clarifications in response, I would be willing to change my score.
For novelty, it is unclear if the results from Lemma 1 to Theorem 1 and 2 are both being stated as novel results. The first part of proof of Theorem 1 is obvious and straightforward, and the other direction has been used before for multiple times as claimed in the paper, what is your novelty exactly here? For the key technical claim of Lemma 1, it looks like this perturbation technique already exists in (Laurent & Brecht, 2018), why do you claim it as a novel argument? 
Besides novelty, there are also some other unclear pieces in this paper needs clarification:
1)	Is the main result which is “no spurious local minima for deep neural network” holds for any differentiable convex loss other than quadratic loss? How will Theorem 1 help us understand the mystery of neural network? 
2)	How does the result help us understand non-linear deep neural network, which is commonly use in practice?
3)	The paper should give some explanations about why the results help training neural networks.