AN ANALYTIC THEORY OF GENERALIZATION DYNAM- ICS AND TRANSFER LEARNING IN DEEP LINEAR NET- WORKS
Much attention has been devoted recently to the generalization puzzle in deep learning: large, deep networks can generalize well, but existing theories bounding generalization error are exceedingly loose, and thus cannot explain this striking performance. Furthermore, a major hope is that knowledge may transfer across tasks, so that multi-task learning can improve generalization on individual tasks. However we lack analytic theories that can quantitatively predict how the degree of knowledge transfer depends on the relationship between the tasks. We develop an analytic theory of the nonlinear dynamics of generalization in deep linear networks, both within and across tasks. In particular, our theory provides analytic solutions to the training and testing error of deep networks as a function of training time, number of examples, network size and initialization, and the task structure and SNR. Our theory reveals that deep networks progressively learn the most important task structure first, so that generalization error at the early stopping time primarily depends on task structure and is independent of network size. This suggests any tight bound on generalization error must take into account task structure, and explains observations about real data being learned faster than random data. Intriguingly our theory also reveals the existence of a learning algorithm that proveably out-performs neural network training through gradient descent. Finally, for transfer learning, our theory reveals that knowledge transfer depends sensitively, but computably, on the SNRs and input feature alignments of pairs of tasks.
This paper builds on the long recent tradition of analyzing deep linear neural networks. In addition to an ample appendix bringing the page total to 20, the authors went over the recommended eight pages, hitting the hard limit of 10 and thus per reviewing directions will be held to a higher standard than the other (mostly 8-page) papers. 
The recent literature on deep linear networks has explored many paths with the hope of producing insights that might help explain the performance of deep neural networks. A recent line of papers by Soudry and Srebro among others focuses on the behavior of stochastic gradient descent. This paper’s analysis comes from a different angle, following the work by Saxe et al (2013) whose analysis considers a (classic one hidden layer) linear teacher network that generates labels and a corresponding student trained to match those labels. The analysis hinges on the singular value decomposition of the composite weight matrix USV^T = W = W^{32} W^{21}.
One aim of the present work, that appears to be a unique contribution above the prior work is to focus on the role played by task structure, suggesting that certain notions of task structure may play a more significant role than architecture and that any bounds which consider architecture but not task structure are doomed to be excessively loose. 
To facilitate their analysis, the authors consider an artificial setup that requires some specific properties. For example, the number of inputs are equal to the input dimension of the network, with the inputs themselves being orthonormal. The labeling function includes a noise term and the singular values of the teacher model admit an interpretation as signal to noise ratios. Given their setup, the authors can express the train and test errors analytically in terms of the weight matrices of the teacher and student and the input-output covariance matrix. The authors then analyze the gradient descent dynamics in what appears to follow the work of Saxe 2013 although I am not an expert on that paper. The analysis focuses on the time dependent evolution of the singular values of the student model, characterized via a set of differential equations.
The next analysis explores a condition that the authors dub “training aligned” initial conditions. This involves initializing the student weights to have the same singular vectors as the training data input-output covariance but with all singular values equal to some amount epsilon. The authors show that the learning dynamics give rise to what they characterize as a singular value “detection wave”. Detecting the modes in descending order by their corresponding singular values.
A set of synthetic experiments show close alignment between theory and experiment.
Section 3.5 offers just one paragraph on a “qualitative comparison to nonlinear networks”. A few issues here are that aesthetically, one-paragraph subsections are not ideal. More problematic is that this theory presumably is building towards insights that might actually be useful towards understanding deep non-linear networks. Since the present material is only interesting as an analytic instrument, I would have hoped for greater emphasis on these connections, with perhaps some hypotheses about the behavior of nonlinear nets driven by this analysis that might subsequently be confirmed or refuted. 
The paper concludes with two sections discussing what happens when nets are trained on randomly labeled data and knowledge transfer across related tasks respectively. 
Overall I think the paper is well-written and interesting, and while I haven’t independently verified every proof, the technical analysis appears to be interesting and sound. The biggest weaknesses of this paper---for this audience, which skews empirical---concern the extent to which the work addresses or provides insight about real neural networks. One potential weakness in this line of work may be that it appears to rely heavily on the linearity of the deep net. While some other recent theories seem more plausibly generalized to more general architectures, it’s not clear to me how this analysis, which hinges so crucially on the entire mapping being expressible as a linear operator, can generalize. 
On the other hand, I am personally of the opinion that the field is in the unusual position of possessing too many tools that “work” and too few new ideas. So I’m inclined to give the authors some license, even if I’m unsure of the eventual utility of the work. 
One challenge in reviewing this paper is that it builds tightly on a number of recent papers and without being an authority on the other works, while it’s possible to assess the insights in this paper, it’s difficult to say conclusively which among them can rightly be considered the present paper’s contributions (vs those of the prior work).