A fundamental goal in deep learning is the characterization of trainability and generalization of neural networks as a function of their architecture and hyperparameters. In this paper, we discuss these challenging issues in the context of wide neural networks at large depths where we will see that the situation simplifies considerably. To do this, we leverage recent advances that have separately shown: (1) that in the wide network limit, random networks before training are Gaussian Processes governed by a kernel known as the Neural Network Gaussian Process (NNGP) kernel, (2) that at large depths the spectrum of the NNGP kernel simplifies considerably and becomes "weakly data-dependent", and (3) that gradient descent training of wide neural networks is described by a kernel called the Neural Tangent Kernel (NTK) that is related to the NNGP. Here we show that in the large depth limit the spectrum of the NTK simplifies in much the same way as that of the NNGP kernel. By analyzing this spectrum, we arrive at a precise characterization of trainability and a necessary condition for generalization across a range of architectures including Fully Connected Networks (FCNs) and Convolutional Neural Networks (CNNs). In particular, we find that there are large regions of hyperparameter space where networks can only memorize the training set in the sense they reach perfect training accuracy but completely fail to generalize outside the training set, in contrast with several recent results. By comparing CNNs with-and without-global average pooling, we show that CNNs without average pooling have very nearly identical learning dynamics to FCNs while CNNs with pooling contain a correction that alters its generalization performance. We perform a thorough empirical investigation of these theoretical results and finding excellent agreement on real datasets.
This paper studies the evolution of Neural Tangent Kernel (NTK) at large-depth regimes. By analyzing the conditional number and eigenvalues, they identify three phases of hyper-parameters; 1) In the chaotic phase NTK converges to an identity matrix, which is easy to train but hardly to generalize. 2) In the ordered phase NTK converges to an all-one matrix, which is hard to train but generalizes well. 3) In the critical phase the conditional number converges to a constant. Furthermore, they also analyze the influence of pooling and flattening in CNNs and identify potential regimes where pooling hurts the generalization. They conduct empirical experiments to supporting their theoretical analyses.
However, I think this paper is worth of more revisions because many theoretical analyses are unjustified. And some potential typos makes the analyses even more difficult to understand.
1) It looks to me that Eq(2) and Eq(6) are contradictory, where T already contains sigma_w and simga_b in Eq(2) but re-multiplied in eq(6).
2) The paper analyzes the dynamics by assuming the variances of inputs are q*, which is debatable. The variance q^l also evolves with the depth increases. It is unclear whether the condition number will change if you takes the evolution of q^l into considered.
3) It is unclear how Eq(9) comes directly from Eq(6), and there aren't any rigorous proofs in the Appendix. Similarly for eq(14).
4) In the paragraph below Eq(11) the paper states that \Theta* becomes an all one-matrix. However, Eq(11) states the diagonal converges to q*/(1-xi_1), but the paragraph below Eq(9) states the off-diagonal converges to q*_{ab}/(1- xi_c). Because q*=q*_{ab} as you stated nearby, do you mean xi_1 = xi_c ? 
5) In the first paragraph of Section 3.3, p^l = q* and p^l=l q*. 
6) In the 2nd contribution, you mentioned "eigenvector correlation", while I cannot find anywhere else introducing this.
7) The plots of Figure 1(b) should behave like convex if the kappa really evolves like x_1^l / l. However it is concave. 
8) In the first experiment, you state "To confirm that the maximal feasible learning rate are ... 2/(lambda_max)". However, learning rates are never discussed in this paper. It is confusing why this experiment is useful. 
Generally speaking, I think the paper needs careful revisions to support its theoretical analyses.