Random Matrix Theory (RMT) is applied to analyze the weight matrices of Deep Neural Networks (DNNs), including both production quality, pre-trained models such as AlexNet and Inception, and smaller models trained from scratch, such as LeNet5 and a miniature-AlexNet. Empirical and theoretical results clearly indicate that the empirical spectral density (ESD) of DNN layer matrices displays signatures of traditionally-regularized statistical models, even in the absence of exogenously specifying traditional forms of regularization, such as Dropout or Weight Norm constraints. Building on recent results in RMT, most notably its extension to Universality classes of Heavy-Tailed matrices, we develop a theory to identify 5+1 Phases of Training, corresponding to increasing amounts of Implicit Self-Regularization. For smaller and/or older DNNs, this Implicit Self-Regularization is like traditional Tikhonov regularization, in that there is a "size scale" separating signal from noise. For state-of-the-art DNNs, however, we identify a novel form of Heavy-Tailed Self-Regularization, similar to the selforganization seen in the statistical physics of disordered systems. This implicit Self-Regularization can depend strongly on the many knobs of the training process. By exploiting the generalization gap phenomena, we demonstrate that we can cause a small model to exhibit all 5+1 phases of training simply by changing the batch size.
This manuscript studies the implicit regularization of neural networks from the perspective of random matrix theory. The authors provide both empirical and theoretical results that aim to show that the empirical spectral density of weights of DNN captures the implicit regularization phenomenon. However, the results are far from rigorous theory and it is not clear how recent results in MP theory yields the statements made in the paper. 
Detailed comments:
1. The empirical studies seem interesting. It seems that two kinds of results are shown. The first one is that ESD fits perfectly for small models, and the second one is that deep models fit heavy-tailed random matrices class. It would be interesting to see more details about how these models are trained, as training greatly affects the value of the weights.
2. Theoretical results are not clearly stated. In section 2, the authors introduce the basics of MP theory. However, it is not clear how to derive the theory in this paper based on the MP theory. It seems that the main theory is the "5+1 phases of training". The definition of these 6 phases is not even explicitly given in this paper. Moreover, the theory of all these phases seems to depend on equation (3) , but there are no lemmas or propositions that gives a rigourious theoretical guarantee.