The weight initialization and the activation function of deep neural networks have a crucial impact on the performance of the training procedure. An inappropriate selection can lead to the loss of information of the input during forward propagation and the exponential vanishing/exploding of gradients during back-propagation. Understanding the theoretical properties of untrained random networks is key to identifying which deep networks may be trained successfully as recently demonstrated by Schoenholz et al. (2017) who showed that for deep feedforward neural networks only a specific choice of hyperparameters known as the 'edge of chaos' can lead to good performance. We complete this analysis by providing quantitative results showing that, for a class of ReLU-like activation functions, the information propagates indeed deeper for an initialization at the edge of chaos. By further extending this analysis, we identify a class of activation functions that improve the information propagation over ReLU-like functions. This class includes the Swish activation, φ swish (x) = x · sigmoid(x), used in Hendrycks & Gimpel (2016), Elfwing et al. (2017) and Ramachandran et al. (2017). This provides a theoretical grounding for the excellent empirical performance of φ swish observed in these contributions. We complement those previous results by illustrating the benefit of using a random initialization on the edge of chaos in this context.
The authors prove some theoretical results under the mean field regime and support their conclusions with a small number of experiments. Their central argument is that a correlation curve that leads to sub-exponential correlation convergence (edge of chaos) can still lead to rapid convergence if the rate is e.g. quadratic. They show that this is the case for ReLU and argue that we must ensure not only sub-exponential convergence, but also have a correlation curve that is close to the identity everywhere. They suggest activation functions that attain conditions as laid out in propositions 4/5 as an alternative.
The paper has many flaws:
- the value of the theoretical results is unclear
- the paper contains many statements that are either incorrect or overly sweeping
- the experimental setup and results are questionnable
Theoretical results:
**Proposition 1: pretty trivial, not much value in itself
**Proposition 2: Pretty obvious to the experienced reader, but nonetheless a valuable if narrow result.
**Proposition 3: Interesting if narrow result. Unfortunately, it is not clear what the ultimate takeaway is. Is quadratic correlation convergence "fast"? Is it "slow"? Are you implying that we should find activation function where at EOC convergence is slower than quadratic? Do those activation functions exist? It would be good to compare this result against similar results for other activation functions. For example, do swish / SeLU etc. have a convergence rate that is less than quadratic?
**Proposition 4: The conditions of proposition 4 are highly technical. It is not clear how one should go about verifying these conditions for an arbitrary activation function, let alone how one could generate new activation functions that satisfy these conditions. In fact, for an arbitrary nonlinearity, verifying the conditions of proposition 4 seems harder than verifying f(x) - x \approx 0 directly. Hence, proposition 4 has little to no value. Further, it is not even clear whether f(x) - x \approx 0 is actually desirable. For example, the activation function phi(x)=x achieves f(x) = x. But does that mean the identity is a good activation function for deep networks? Clearly not.
**Proposition 5: The conditions of prop 5 are somewhat simpler than those of prop 4, but since we cannot eliminate the complicated condition (ii) from prop 4, it doesn't help much.
**Proposition 6: True, but the fact that we have f(x) - x \approx 0 for swish when q is small is kind of obvious. When q is small, \phi_swish(x) \approx 0.5x, and so swish is approximately linear and so its correlation curve is approximately the identity. We don't need to take a detour via propposition 4 to realize this.
Presentation issues:
- While I understand the point figures 1, 2 and 4b are trying to make, I don't understand what those figures actually depict. They are insufficiently labeled. For example, what does each axis represent?
- You claim that for ReLU, EOC = {(0,\sqrt{2})}. But this is not true. By definition 2, EOC is a subset of D_\phi,var. But {(0,\sqrt{2})} is not in D_\phi,var, because it simply leaves all variances unchanged and does not cause them to converge to a single value. You acknowledge this by saying "For this class of activation functions, we see (Proposition 2) that the variance is unchanged (qal = qa1) on the EOC, so that q does not formally exist in the sense that the limit of qal depends on a. However,this does not impact the analysis of the correlations." Section 2 is full of complicated definitions and technical results. If you expect the reader to plow through them all, then you should really stick to those definitions from then on. Declaring that it's fine to ignore your own definitions at the beginning of the very next section is bad presentation. This problem becomes even worse in section 3.2, where it is not clear which definition is actually used for EOC in your main result (prop 4), making prop 4 even harder to parse than it already is.
Correctness issues:
- "In this chaotic regime, it has been observed in Schoenholz et al. (2017) that the correlations converge to some random value c < 1" Actually, the correlation converges deterministically, so c is not random.
- "This means that very close inputs (in terms of correlation) lead to very different outputs. Therefore, in the chaotic phase, the output function of the neural network is non-continuous everywhere." Actually, the function computed by a plain tanh network is continuous everywhere. I think you mean something like "the output can change drastically under small changes to the input". But this concept is not the same as discontinuity, which has an established formal definition.
- "In unreported experiments, we observed that numerical convergence towards 1 for l ≥ 50 on the EOC." Covergence of a sequence is a property of the limit of the sequence, and not of the 50th element. This statement makes no sense. Also if you give a subjective interpretation of those experimental results, you should present the actual results first.
- "Tanh-like activation functions provide better information flow in deep networks compared to ReLU-like functions." This statement is very vague and sweeping. Also, one could argue that the fact that ReLU is much more popular and tends to give better results than tanh in practice disproves the statement outright.
- "Tanh-like activation functions provide better information flow in deep networks compared to ReLU-like functions. However, these functions suffer from the vanishing gradient problem during back-propagation" At the edge of chaos, vanishing gradients are impossible! As Schoenholz showed, at the edge of chaos, \chi_1=1, but \chi_1 is also the rate of growth of the gradient. Pascanu et al (2013) discussed vanishing gradients in RNNs, which is a different story.
- "Other activation functions that have been shown to outperform empirically ReLU such as ELU (Clevert et al. (2016)), SELU (Klambauer et al. (2017)) and Softplus also satisfy the conditions of Proposition 4 (see Supplementary Material for ELU)." Firstly, SeLU does not satisfy proposition 4. f(x) \approx x requires \phi to be close to a linear function in the range where the pre-activations occur. Since SeLU has a kink at 0, it cannot be close to a linear function no matter how small the pre-activations are. Secondly, softplus also doesn't satisfy proposition 4, as \phi(0) = 0 does not hold. Thirdly, this statement is too sweeping. If ELU / SELU / Softplus "outperform" ReLU, why is ReLU still used in practice? At best, those nonlinearities have been shown to outperform in a few scenarios.
- "We proved in Section 3.2 that the Tanh activation guarantees better information propagation through the network when initialized on the EOC." Prop 4 only applies in the limit as \sigma_b converges to 0. So you can't claim that you showed tanh as "better information propagation" in general.
- "However, for deeper networks (L ≥ 40), Tanh is stuck at a very low test accuracy, this is due to the fact that a lot of parameters remain essentially unchanged because the gradient is very small." But in figure 6b the accuracy for tanh is decreasing rapidly, so therefore the parameters are not remaining "essentially unchanged", as this would also cause the accuracy to remain essentially unchanged. Also, if the parameter changes are too small ... why not increase the learning rate?
- "To obtain much richer priors, our results indicate that we need to select not only parameters (σb , σw ) on the EOC but also an activation function satisfying Proposition 4." Prop 4 only applies when \sigma_b is small, so you additionally need to make sure \sigma_b small.
- "In the ordered phase, we know that the output converges exponentially to a fixed value (same value for all Xi), thus a small change in w and b will not change significantly the value of the loss function, therefore the gradient is approximately zero and the gradient descent algorithm will be stuck around the initial value." But you are using Adam, not gradient descent! Adam explicitly corrects for this kind of gradient vanishing, so a small gradient can't be the reason for the lack of training success.
Experimental issues:
- "We use the Adam optimizer with learning rate lr = 0.001." You must tune the learning rate independently for each architecture for an ubiased comparison.
- In figure 6b, why does tanh start with a high accuracy and end up with a low accuracy? I've never seen a training curve like this ... This suggests something is wrong with your setup.
- You should run more experiments with a larger variety of activation functions.
Minor comments: 
- "Therefore, it is easy to see that for any (σb , σw ) such that F is increasing and admits at least one fixed point,wehaveKφ,corr(σb,σw) ≥ qwhereqistheminimalfixedpoint;i.e. q := min{x : F(x) = x}." I believe this statement is true, but I also think it requires more justification.
- At the end of page 3, I think \epsilon_r should be \epsilon_q
There are some good ideas here, but they need to be developed/refined/polished much further before publication. The above (non-exhaustive) list of issues will hopefully be helpful for this.
### Addendum ###
After an in-depth discussion with the authors (see below), my opinion on the paper has not changed. All of my major criticisms remain: (1) There are far easier ways of achieving f(x) ~ x than propositions 4/5/7, i.e. we simply have to choose \phi(x) approximately linear. (2) The experiments are too narrow, and learning rates are badly chosen. (3) The authors do not discuss the fact that as f(x) gets too close to x, performance actually degrades as \phi(x) gets too close to a linear function. (Many other criticisms also remain.)
The one criticism that the authors disputed until the end of the discussion is criticism (1). Their argument seems to hinge on the fact that their paper provides a path to construct activation function that avoid "structural vanishing gradients", which they claim 'tanh' suffers from. While they acknowledge that tanh does not necessarily suffer from "regular" vanishing gradients (as shown by "Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice" and "Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks"), they claim it suffers from structural vanishing gradients. I do not believe that there is such a thing as structural vanishing gradients. However, even if such a concept did exist, it falls on the the authors to provide a clear definition / explanation, which they neither do in the paper nor the rebuttal.