PCA of high dimensional random walks with comparison to neural network training
One technique to visualize the training of neural networks is to perform PCA on the parameters over the course of training and to project to the subspace spanned by the first few PCA components. In this paper we compare this technique to the PCA of a high dimensional random walk. We compute the eigenvalues and eigenvectors of the covariance of the trajectory and prove that in the long trajectory and high dimensional limit most of the variance is in the first few PCA components, and that the projection of the trajectory onto any subspace spanned by PCA components is a Lissajous curve. We generalize these results to a random walk with momentum and to an Ornstein-Uhlenbeck processes (i.e., a random walk in a quadratic potential) and show that in high dimensions the walk is not mean reverting, but will instead be trapped at a fixed distance from the minimum. We finally analyze PCA projected training trajectories for: a linear model trained on CIFAR-10; a fully connected model trained on MNIST; and ResNet-50-v2 trained on Imagenet. In all cases, both the distribution of PCA eigenvalues and the projected trajectories resemble those of a random walk with drift.
The paper looks at the dynamics of training in DNNs through the lens of PCA in comparison with random walk. The work first looks at the behavior of first PCA components in random walk and compares it with the OU process. Then it looks at the DNNs and finds out that in the early phases of the training the behavior of weights are ballistic and later on they resemble more of an OU like process. The DNN part of the study is quite empirical but derivations are provided for the random walk.  Recently, there have been several works on understanding the landscape of loss functions and dynamics on it. The flatness of the minima may imply diffusive behavior at the bottom of the landscape which is consistent with the findings in this paper as well (generalization connection: https://arxiv.org/abs/1609.04836, local geometry at the bottom (for instance discussion in 298-304 fits into this framework): https://arxiv.org/abs/1803.06969, connecting solutions at the bottom: https://arxiv.org/abs/1806.06977, diffusive (on non-glassy loss surface) dynamics at the bottom: https://arxiv.org/abs/1803.06969, PCA on NNs: https://arxiv.org/abs/1602.07320, etc...). It's pity that the current work doesn't discuss the current literature, but well, ML research these days can lack many things including scrutiny in experiments and I rather see a methodologically sound work than the one with missing core discussion.   Speaking of the discussion, the paragraph between lines 22-30 has the correct conclusion, but the two papers cited there actually give different take-home messages. Dauphin et al is worried about the saddle points problem, and Choromanska et al is worried that the landscape will be mapped to a Hamiltonian of a spin glass (which is not true). The empirical results on finding the same loss values no matter what is interesting, but similar experiments can be found in several other works that do not necessarily claim that the landscape is glassy.   Regarding the claims of the referred papers in lines 48-51, all three papers need to use a version of the functional central limit theorem which is very hard to justify in the case of DNN where the number of parameters is huge (for example, for reasons mentioned in lines 206-209!). This debate on whether SGD is Langevin type dynamics is a major debate and it is quite unclear that this is the case. It's another drawback of the paper that it just cites a few works for which that support their vague idea without any discussion in depth. Overall, I think the introduction and discussion part of the paper is quite poor and it can be misleading without precisely stating such pitfalls.   The point of the above comments is not to include a massive discussion on every area, but to reduce the certainty in the language by picking a few papers that fit the general discussion. The paper has only 12 citations and 5 of them are quite non-essential as I have discussed above. At this stage of ML research, this is acceptable only if the work shows what they find. It is totally OK to show the results of their findings with minimal discussion, and the work will have its own value as a solid empirical work which is quite interesting and will have an impact in the community. But it is not OK to attempt to have a discussion with the missing sides of the story.    A further note on experiments: It is appreciated that the experiments are carried on in large-scale models with real data. However, it would be great to see it in a small fully connected model, as well. Say on MNIST and 2 hidden layer fully connected model, how wide should the network be in order to see this behavior consistently? If it is claimed to work on ResNets, there must be a regime for which it works on FCs. Another advantage of such a model is that one won't need to pick a layer in the middle and can do PCA on the full weight vector.   Also, for the ResNet, why only the sixth convolutional layer? Do you see this behaviour consistently in early layers and final layers as well? If not, why do you think that this is the case?   Overall, I think the direction of the paper is great but it would have been much richer with further discussions, it would've been less misleading with fewer discussions. It would've been more complete with more experiments even if, say, the claim doesn't work for the last layers, for example. It is a good paper, but I think there is room for crucial improvement in the discussion section (either by expansion or removal), and further room for intermediate complexity experiments. (And of course, it would be great to have further information on how exactly the experiments are conducted for reproducibility...)