Which Neural Net Architectures Give Rise to Exploding and Vanishing Gradients?
We give a rigorous analysis of the statistical behavior of gradients in a randomly initialized fully connected network N with ReLU activations. Our results show that the empirical variance of the squares of the entries in the input-output Jacobian of N is exponential in a simple architecture-dependent constant , given by the sum of the reciprocals of the hidden layer widths. When is large, the gradients computed by N at initialization vary wildly. Our approach complements the mean field theory analysis of random networks. From this point of view, we rigorously compute finite width corrections to the statistics of gradients at the edge of chaos.
The paper studies the gradient vanishing/exploding problem (EVGP) theoretically in deep fully connected ReLU networks. As a substitute for ensuring if gradient vanishing/exploding has been avoided, the paper proposes two criteria: annealed EVGP and quenched EVGP. It is finally shown that both these criteria are met if the sum of reciprocal of layer widths of the network is a small number (thus the width of all layers should ideally be large). To confirm this empirically, the paper uses an experiment from a concurrent work.  Comments:  To motivate formally studying EVGP in deep networks, the authors refer to papers which suggest looking at the distribution of singular values of the input-output Jacobian. If the singular values are far from 1 and take both small and large values, it implies the network is suffering from exploding gradient or vanishing gradient problem or both. However, to study the Jacobian analytically, the authors propose to study the annealed and quenched criteria, which in some sense measure how much fluctuation the Jacobian has across its elements. The analysis suggests a high fluctuation is bad while small fluctuation avoids EVGP.   If I understand the paper correctly, my main concern with the proposed criteria is that having a small fluctuation in the Jacobian across its elements would mean all elements are similar. In this case, all but one sigular values of the Jacobian would collapse to ~0, which is bad. So the proposed criteria seems problematic to me.  Regarding the cited experiment for an empirical justification of the proposed theory, the paper from which the figure is taken proposes an alternative theoretical reason for this behavior. This makes it hard to understand whether a more appropriate reason for this behavior is the justification in the original paper or this paper.  The paper is clearly written and original in the sense that it proposes to study EVGP but may be problematic.  Comments after reading the rebuttal:  I am still not sure I understand how the input-output Jacobian Z is well conditioned (singular values are close to one) if each entry of the Jacobian Z_pq^2 has a finite expected value (Theorem 1 Eq 10) and variance as shown in Eq 11 (of course these results are extended to talk about the empirical variance of the squared entries of Z in theorem 2 and 3). For instance, consider Z to be of dimension 400x800. If I generate the entries of Z by sampling from a gaussian with mean 1/400 and variance 1/400 (I have assumed n0=400 and width of the network layers to be infinite in theorem 1 to roughly use these numbers), I get a matrix Z that has singular values spanning the range 0.02 to 1.4 (thus condition number 1.4/0.02=70>>1).  Another point I would like to raise is that even if the input-output Jacobian is well conditioned, it is essentially a product of terms like d h_t/d h_{t-1}, where h_t is the hidden state at depth t. Now I can have a matrix Z that is even identity, but is the result of the product of two matrices as Z = AB, where A has very large singular values and B has the inverse singular values of A, thus resulting in the singular values of 1 for Z. Here A and B signify partial derivatives like d h_t/d h_{t-1} for 2 consecutive layers. As can be seen, the gradient for layer corresponding to A are exploding while that of B will are vanishing.  I feel these aspects need to be made clear in the paper.