The critical locus of the loss function of a neural network is determined by the geometry of the functional space and by the parameterization of this space by the network's weights. We introduce a natural distinction between pure critical points, which only depend on the functional space, and spurious critical points, which arise from the parameterization. We apply this perspective to revisit and extend the literature on the loss function of linear neural networks. For this type of network, the functional space is either the set of all linear maps from input to output space, or a determinantal variety, i.e., a set of linear maps with bounded rank. We use geometric properties of determinantal varieties to derive new results on the landscape of linear networks with different loss functions and different parameterizations.
The authors use tools from algebraic geometry to study the critical points that arise in linear neural networks under a variety of loss functions.  In particular, they study linear maps that are parameterized (or overparameterized) as the product of multiple matrices.  The authors introduce a distinction between two types of critical points (for the linear function composed with a provided loss): there are those that are due to the actual manifold of functions that are provided to the loss (pure critical points), and there are those that are due to parameterizations of this manifold (spurious critical points).  The authors show that in the case that the linear function has maximal rank (is "filling"), then there are no non-global minima in the loss landscape.  In the case of lower rank ("nonfilling") linear functions, this property does not hold generally, but does hold for quadratic losses.   The authors show that for this framework, there can be exponentially many disconnected components in parameter space corresponding to global minima.  
This paper provides important insight into the loss landscape of neural networks, and it exhibits nuanced observations even in the linear case.  Naturally, there are many important follow-up studies that can be done in the case of non-linear networks.  It is vital to have a thorough understanding before that work can be satisfactorily undertaken.
It would benefit the authors to provide more summary/commentary in the main results section.  Currently, that section comes across as a list of lemmas, theorems, propositions, and corollaries, with relatively little narration in between.  It was challenging for me to identify which of these items were the main results, as summarized on in the first paragraph of the conclusions section.  I recommend that any theorem/corollary/proposition that is the primary basis for a claim made in the conclusion should receive significantly more commentary in Section 3.
Minor comment:
In the first paragraph of the conclusion, it says that "... the absence of ... is a rather general property ...".  The word "rather" should be clarified.