GRADIENT DESCENT ALIGNS THE LAYERS OF DEEP LINEAR NETWORKS
This paper establishes risk convergence and asymptotic weight matrix alignment -a form of implicit regularization -of gradient flow and gradient descent when applied to deep linear networks on linearly separable data. In more detail, for gradient flow applied to strictly decreasing loss functions (with similar results for gradient descent with particular decreasing step sizes): (i) the risk converges to 0; (ii) the normalized i th weight matrix asymptotically equals its rank-1 approximation u i v i ; (iii) these rank-1 matrices are aligned across layers, meaning |v i+1 u i | → 1. In the case of the logistic loss (binary cross entropy), more can be said: the linear function induced by the network -the product of its weight matrices -converges to the same direction as the maximum margin solution. This last property was identified in prior work, but only under assumptions on gradient descent which here are implied by the alignment phenomenon.
In this work the authors prove several claims regarding the inductive bias of gradient descent and gradient flow trained on deep linear networks with linearly separable data. They show that asymptotically gradient descent minimizes the risk, each weight matrix converges to its rank one approximation and the top singular vectors of two adjacent weight matrices align. Furthermore, for the logistic and exponential loss the induced linear predictor converges to the max margin solution. 
This work is very interesting and novel. It provides a comprehensive and exact characterization of the dynamics of gradient descent for linear networks. Such strong guarantees are essential for understanding neural networks and extremely rare in the realm of non-convex optimization results. The work is a major contribution over the paper of Gunasekar et al. (2018) which assume that the risk is minimized. The proof techniques are interesting and I believe that they will be useful in analyzing neural networks in other settings.
Regarding Lemma 3, the proof is not clear. Lemma 8 does not exist in the paper of Soudry et al. (2017). It is also claimed that with probability 1 there are at most d support vectors. How does this relate with assumption 3, which implies that there are at least d support vectors?
-------Revision---------
Thank you for the response. I have not changed the original review.