Critical initialisation for deep signal propagation in noisy rectifier neural networks
Stochastic regularisation is an important weapon in the arsenal of a deep learning practitioner. However, despite recent theoretical advances, our understanding of how noise influences signal propagation in deep neural networks remains limited. By extending recent work based on mean field theory, we develop a new framework for signal propagation in stochastic regularised neural networks. Our noisy signal propagation theory can incorporate several common noise distributions, including additive and multiplicative Gaussian noise as well as dropout. We use this framework to investigate initialisation strategies for noisy ReLU networks. We show that no critical initialisation strategy exists using additive noise, with signal propagation exploding regardless of the selected noise distribution. For multiplicative noise (e.g. dropout), we identify alternative critical initialisation strategies that depend on the second moment of the noise distribution. Simulations and experiments on real-world data confirm that our proposed initialisation is able to stably propagate signals in deep networks, while using an initialisation disregarding noise fails to do so. Furthermore, we analyse correlation dynamics between inputs. Stronger noise regularisation is shown to reduce the depth to which discriminatory information about the inputs to a noisy ReLU network is able to propagate, even when initialised at criticality. We support our theoretical predictions for these trainable depths with simulations, as well as with experiments on MNIST and CIFAR-10. ‡
This paper analyzes signal propagation in vanilla fully-connected neural networks in the presence of noise. For ReLU networks, it concludes that the initial weight variance should be adjusted from the "He" initialization to account for the noise scale. Various empirical simulations corroborate this claim.  Generally speaking, I believe studying signal propagation in random neural networks is a powerful way to build better initialization schemes and examining signal propagation in the presence of noise is an interesting direction. The paper is well-written and easy to read. However, I found the analysis and conclusions of this work to be rather weak and I do not believe it is appropriate for publication at NIPS at this time.  The technical contributions of this work are readily derived from eqn. (5), which is a fairly simple modification of the recursion relation studied in [1,2]. From eqn. (5), it is easy to see that multiplicative noise can be absorbed into \sigma_w, and additive noise can be absorbed into \sigma_b. While this is a useful observation, to me it seems fairly trivial from the technical perspective and insufficient to serve as the backbone of the entire paper.  Perhaps more importantly, the paper focuses entirely on "q", i.e. pre-activation variances, and does not analyze the propagation of "c", i.e. pre-activation correlations. The latter is necessary for understanding the phase diagram, the stability of fixed points, and the trainable depth scales. Even if the phase diagram ends up looking identical to the noise-free case (modulo redefinitions of \sigma_w and \sigma_b to account for the noise), this point should have been discussed and investigated.  The trainability experiments in Figure 4 go out to depth 200. While this is certainly quite deep, it may not be deep enough to demonstrate the breakdown of trainability arising from ill-conditioned Jacobians. Indeed, in [3], critically-initialized networks with ill-conditioned Jacobians can train out to depth 200, albeit much more slowly than their well-conditioned counterparts.  Overall, while I like the general idea behind this paper, I do not find the results significant enough to merit publication at NIPS at this stage.  [1] Poole, Ben, et al. "Exponential expressivity in deep neural networks through transient chaos." Advances in neural information processing systems. 2016. [2] Schoenholz, Samuel S., et al. "Deep Information Propagation." ICLR 2017. [3] Pennington, Jeffrey, Samuel Schoenholz, and Surya Ganguli. "Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice." Advances in neural information processing systems. 2017.  Edit: After reviewing the authors' response and examining the extensive new experiments, I have increased my score to 7 and vote for acceptance. The authors added new analysis on the phase diagram, convergence rates of correlations, and many other important metrics. I believe these analyses will significantly improve the paper. I still think the technical meat of the paper is rather weak, but given the extensive new experiments I think the paper is worthy of publication.