How to Start Training: The Effect of Initialization and Architecture
We identify and study two common failure modes for early training in deep ReLU nets. For each, we give a rigorous proof of when it occurs and how to avoid it, for fully connected, convolutional, and residual architectures. We show that the first failure mode, exploding or vanishing mean activation length, can be avoided by initializing weights from a symmetric distribution with variance 2/fan-in and, for ResNets, by correctly scaling the residual modules. We prove that the second failure mode, exponentially large variance of activation length, never occurs in residual nets once the first failure mode is avoided. In contrast, for fully connected nets, we prove that this failure mode can happen and is avoided by keeping constant the sum of the reciprocals of layer widths. We demonstrate empirically the effectiveness of our theoretical results in predicting when networks are able to start training. In particular, we note that many popular initializations fail our criteria, whereas correct initialization and architecture allows much deeper networks to be trained.
The paper studies two failure modes during the initial phase of deep network training: 1. The mean length of activation vector in the last layer increases/decreases exponentially with depth. The mean is taken over the randomness in weights/biases while the input is considered fixed. 2. The variance of length of activation vector grows exponentially with depth.  It has been shown that FM1 in fully connected architectures can be avoided by sampling the weights of the network from a symmetric distribution with variance 2/fan-in. For ResNets, it has been shown that in addition to weight initialization, each residual block needs to be scaled appropriately such that the sum of reciprocal of these scaling factors is close to 1 (the latter is not mentioned explicitly/clearly).  It is shown that FM2 is avoided for fully connected networks when using large width in networks. More specifically, the variance in the length of activation vector is exponential in the sum of reciprocal of widths of layers. For ResNets it is shown that addressing FM1 automatically rules out FM2.   All theoretical results are complemented side-by-side with appropriate experiments. For FM1, the mean length of activations is shown with depth for different initializations along with the number of epochs needed to achieve 20% performance. For FM2, only the latter is shown (It would be informative to also show how the empirical variance behaves for different cases).  Comments:  FM1 is well studied in literature for fully connection architecture as acknowledged by authors as well. The novelty seems to be twofold: 1. FM1 has not been well studied for ResNet architecture. It is studied in this paper. 2. When studying FM1, previous literature assumes the input is random and the variance of activations is calculated over both the randomness in input and parameter initialization distribution. In this paper, input is considered fixed.  Regarding point 2 above, I am not sure I understand why preserving the mean length of activation vector in the last layer for a *fixed input* is important. In previous work, the motivation is to preserve the information content in the input. As a proxy, this information is measured by the variance in input and so calculating the variance of hidden units over the input/parameter distribution makes sense. The reasoning for considering fixed input is not discussed in the paper. Please clarify.  Also, since the theory predicts that FM2 in fully connected networks can be avoided by using network widths such that the sum of reciprocal is small, I am curious if it equivalent to use something like widths 10,20,30,40 vs widths 40,30,20,10 in a network. In these two cases, the sum of reciprocal of widths is identical, but one has width increasing with depth while the other has width decreasing with depth. My intuition is that the latter may have a poor "mean epochs to 20%". I recommend the authors to add this experiment. If the results are unexpected, it would be interesting to leave it as an open thread, and in the other case it would make the theory more compelling.  The authors should acknowledge in line 173 that the paper "A Closer Look at Memorization in Deep Networks" discussed (their Figure 6) that time to convergence is better for wider networks.  (I did not go through the proofs.)  Clarity:  The novelty of the paper is not stated clearly. For instance, the first two contributions mentioned in the paper are already known. The novelty in these contributions is not clear.  There are two problems in the way the authors have introduced FM1 and FM2:  1. Clearly identifying FM1 and FM2 in previous work and better motivating why they (can) cause failure is very important and does not seem to be done properly. Instead, after stating FM1 and FM2 in the introduction, the rest of the paper discusses how to solve it. This is especially the case with FM2 which is not well known. 2. While it is stated in the definition of FM1 and FM2 that quantities increase/decrease exponentially with depth, and that it is problematic, it is never stated explicitly how these quantities should ideally behave: mean length should be preserved and variance should be as small as possible.    Originality:  As mentioned above, the novelty needs to be stated clearly and the motivation for FM2 and why FM1 is considered for fixed input needs to be clarified.  Overall, I think the paper is somewhat original. The drawback is that initializing networks parameters from distributions with variance 2/fan-in is already known. But the part about ResNet is novel. The part about why large width is important is also novel and very interesting.  Significance: I think the intialization scheme for ResNet (scaling residual blocks seems to be the novel part) can be of importance for practitioners.   I have given an overall score of 6 mainly because of lack of clarity in the paper.  ## Post rebuttal update:  The authors clarified some of the questions I raised. Although the experiment on training a network with decreasing width is not conducted for the rebuttal as I proposed, the authors agree to include it in the final submission.   However I have two other concerns:  1. The paper talks about preserving activation length and variance which was originally proposed as a way to preserve information during the forward pass of a deep network. However there is no discussion on gradient norm preservation which is an important for avoiding vanishing/exploding gradients problem during initialization and is also important for avoiding the initial failure during training.  2. The constant C that appears in theorem 5 is only discussed to the point that it exists and is positive. But no further discussion is provided. This is a concern because a very large C satisfying theorem 5 would be trivial since the activation length will effectively not be preserved. This is further important because the theorem talks about any fixed input vector. When considering a large number of input vectors (a dataset), it is more likely for the variance in activation norm is large for some of the samples.  For these reasons, I am sticking with my original score.