We analyze the joint probability distribution on the lengths of the vectors of hidden variables in different layers of a fully connected deep network, when the weights and biases are chosen randomly according to Gaussian distributions, and the input is in {−1, 1} N . We show that, if the activation function φ satisfies a minimal set of assumptions, satisfied by all activation functions that we know that are used in practice, then, as the width of the network gets large, the "length process" converges in probability to a length map that is determined as a simple function of the variances of the random weights and biases, and the activation function φ. We also show that this convergence may fail for φ that violate our assumptions.
second layer may not be independent, even for some permissible φ like the ReLU. The results of this section contradict claims made in [14,10]. Section 5 describes some simulation experiments verifying some of the findings of the paper, and illustrating the dependence among the values of the hidden nodes.
Our analysis of the convergence of the length map borrows ideas from Daniely, et al. [2], who studied the properties of the mapping from inputs to hidden representations resulting from random Gaussian initialization. Their theory applies in the case of activation functions with certain smoothness properties, and to a wide variety of architectures. Our analysis treats a wider variety of values of σ w and σ b , and uses weaker assumptions on φ.
Summary: the paper proves the convergence of empirical length map (length process) in NN to the length map for a permissible activation functions in a wide-network limit. The authors also show why the assumptions on the permissible functions can not be relaxed.
Quality: the paper seems to be technically correct. However, the authors do not discuss any consequence of their result. Why was it important to prove it? What does it tell us about the networks? While the proof may be of interest to the authors of [14] to correct their (possible) mistakes, I think the paper will go under the radar for most people and thus encourage the authors to heavily revise the paper.
Clarity: the writing is clear in general. The proofs sometimes jump over non-trivial things and explain easy steps, but that maybe subjective. The paper spends no effort explaining the contribution and its consequences.
Originality: the proven statements are novel and extend/fix the claims of [14]
Significance: as said above, I believe that in the current form the paper will have little to no impact. The importance of proving the main statement under more general conditions on activation functions is doubtful and the authors do not comment on that.
Minor comments:
* when introducing T{i,:,:} the <> notation is not clear. I could guess it from the later usage of the symbol, but these brackets can mean a lot of things, e.g. bracket mean (Section 2.1)
* it would be beneficial to define the main objects, wide-network limits, in a more formal way (Section 2.3)
* how the wide regime (large N) is interesting for studying deep NNs? [14] discusses that to some extent, but this should be explained here as well
* q_0 is never defined
* it's good practice to add numbers to all equations
* I believe the claim in the appendix of [14] was meant to be conditionally independent (see also the reviews of [14]). It's clear that preactivations should not be independent and, while technically interesting, spending a page of theorem 10 and on plots seems unnecessary. Even in the paper's example preactivations are uncorrelated in the limit of large N.
* I don't see the point of having experiments in this paper. The authors have already proven the fact. Also, it is not clear how to read the plots (no axes, little description) and come to the statements from page 9.
********************
After the authors' response:
If the main motivation of the paper is to fix the mistakes in [14], then the paper should clearly state so, in addition to explaining why fixing is necessary. While I believe that pointing out other paper's mistakes and correcting them is important, the current state of the paper leads me to keeping my initial score and recommending to reject the paper.