Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
It is well known that weight initialization in deep networks can have a dramatic impact on learning speed. For example, ensuring the mean squared singular value of a network's input-output Jacobian is O(1) is essential for avoiding exponentially vanishing or exploding gradients. Moreover, in deep linear networks, ensuring that all singular values of the Jacobian are concentrated near 1 can yield a dramatic additional speed-up in learning; this is a property known as dynamical isometry. However, it is unclear how to achieve dynamical isometry in nonlinear deep networks. We address this question by employing powerful tools from free probability theory to analytically compute the entire singular value distribution of a deep network's input-output Jacobian. We explore the dependence of the singular value distribution on the depth of the network, the weight initialization, and the choice of nonlinearity. Intriguingly, we find that ReLU networks are incapable of dynamical isometry. On the other hand, sigmoidal networks can achieve isometry, but only with orthogonal weight initialization. Moreover, we demonstrate empirically that deep nonlinear networks achieving dynamical isometry learn orders of magnitude faster than networks that do not. Indeed, we show that properly-initialized deep sigmoidal networks consistently outperform deep ReLU networks. Overall, our analysis reveals that controlling the entire distribution of Jacobian singular values is an important design consideration in deep learning.
The article is focused on the problem of understanding the learning dynamics of deep neural networks depending on both the activation functions used at the different layers and on the way the weights are initialized. It is mainly a theoretical paper with some experiments that confirm the theoretical study. The core of the contribution is made based on the random matrix theory. In the first Section, the paper describes the setup -- a deep neural network as a sequence of layers -- and also the tools that will be used to study their dynamics. The analysis mainly relies on the study of the singular values density of the jacobian matrix, this density being computed by a 4 step methods proposed in the article. Then this methodology is applied for 3 different architectures: linear networks, RELU networks and tanh networks, and different initialization of the weights -- Ginibre ensemble and Circular Real Ensemble. The theoretical results mainly show that tanh and linear networks may be trained efficiently -- particularly using orthogonal matrix initialization -- while RELU networks are poorly conditioned for the two initialization methods. The experiments on MNIST and CIFAR-10 confirm these results. 
First, I must say that the paper is a little bit out of my main research domain. Particularly, I am not familiar with random matrix theory. But, considering that I am not aware of this type of article, I find the paper well written: the focus is clear, the methodology is well described, references include many relevant prior works helping to understand the contribution, and the authors have also included an experimental section. Concerning the theoretical part of the paper, the proposed results seem correct, but would certainly need a deeper checking by an expert of the domain. Concerning the experimental section, the experiments confirm the theoretical results. But I must say that I am a little bit skeptical concerning the topic of the paper since all the demonstrations are focused on the simple deep neural network architectures with linear+bias transformations between the layers. It is a little bit surprising to see results on CIFAR-10 which is typically a computer vision benchmark where convolutional models are used, which is not the case here. So I have some doubt concerning the relevance of the paper w.r.t existing architectures that are now much more complex than sequences of layers. On another side, this paper is certainly a step toward the analysis of more complex architectures, and the author could argue on how difficult it could/will be to use the same tools (random matrix theory) to target for example convolutional networks.