A critical part of the training process of neural networks takes place in the very first gradient steps post-initialization. In this work, we study the connection between the network's architecture and initialization parameters, to the statistical properties of the gradient in random fully connected ReLU networks, through the study of the the per layer Jacobian in finite sized networks. We compare three types of architectures: vanilla networks, ResNets and DenseNets. In our analysis, we show that while the variance of Jacobian squared norm is exponential in depth for ResNets, and polynomial for DenseNets, there exists an initialization strategy for both, such that the norm is preserved through arbitrary depths, preventing exploding or decaying gradients in deep networks.We also show that the statistics of the per layer Jacobian norm is a function of the architecture and the layer's size, but surprisingly, not the layer's depth.
This paper analyzes the statistics of activation norms and Jacobian norms for randomly-initialized ReLU networks in the presence (and absence) of various types of residual connections. Whereas the variance of the gradient norm grows with depth for vanilla networks, it can be depth-independent for residual networks when using the proper initialization.
The main theoretical results stem from a norm propagation duality that allows computations of certain statistics using simplified architectures. As far as I know, this is a novel way of analyzing multi-pathway architectures, and the method and the implied results will be of interest to deep-learning theorists.
There are two types of empirical results, one to verify the theoretical predictions, and the other to see if those predictions translate into improved generalization performance on some real-world tasks. Figures 2 and 3 show good agreement with theory. I am less convinced by Fig 4. Unless I missed something, I thought the implication from the theoretical analysis was that multi-pathway models should behave better than single-pathway models at large depth, and for single-pathway models concatenated ReLU may have slightly better large-depth behavior than standard ReLU. Doesn't this suggest that for large depth DenseNet ~= ResNet > CR > ReLU? Is the good performance of CR supposed to be understood as supporting evidence for the theoretical analysis? Or is the point more like "the theory suggests CR might be interesting, and indeed it performs well, but we leave the details of generalization performance to a future study"? It would be important to specify the desired interpretation here.
But I think regardless of the interpretation, the empirical results are not strong enough to stand on their own as support for CR, especially since this type of architecture has been introduced and studied previously (c.f. also the looks-linear setup from [1]). Therefore my evaluation is mostly based on the theoretical contributions, which I think are themselves are probably sufficient for publication.
One main area for improvement is in the related work. A significant line of work is omitted that studies higher-order statistics of Jacobian matrices in the large-depth, infinite-width regime [2-8], and while this is at infinite width, it work for any non-linearity, so is an important point for comparison. Other kinds of initialization schemes are also relevant for comparison, e.g. [9-10].
[1] Balduzzi, David, et al. "The shattered gradients problem: If resnets are the answer, then what is the question?." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.
[2] Pennington, Jeffrey, Samuel Schoenholz, and Surya Ganguli. "Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice." Advances in neural information processing systems. 2017.
[3] Pennington, Jeffrey, Samuel Schoenholz, and Surya Ganguli. "The emergence of spectral universality in deep networks." International Conference on Artificial Intelligence and Statistics. 2018.
[4] Hayase, Tomohiro. "Almost Surely Asymptotic Freeness for Jacobian Spectrum of Deep Network." arXiv preprint arXiv:1908.03901 (2019).
[5] Tarnowski, Wojciech, et al. "Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function." The 22nd International Conference on Artificial Intelligence and Statistics. 2019.
[6] Burkholz, Rebekka, and Alina Dubatovka. "Initialization of ReLUs for Dynamical Isometry." arXiv preprint arXiv:1806.06362 (2018).
[7] Xiao, Lechao, et al. "Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks." International Conference on Machine Learning. 2018.
[8] Chen, Minmin, Jeffrey Pennington, and Samuel Schoenholz. "Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks." International Conference on Machine Learning. 2018.
[9]Sutskever, Ilya, et al. "On the importance of initialization and momentum in deep learning." International conference on machine learning. 2013.
[10] Le, Quoc V., Navdeep Jaitly, and Geoffrey E. Hinton. "A simple way to initialize recurrent networks of rectified linear units." arXiv preprint arXiv:1504.00941 (2015).