We provide a theoretical explanation for the great performance of ResNet via the study of deep linear networks and some nonlinear variants. We show that with or without nonlinearities, by adding shortcuts that have depth two, the condition number of the Hessian of the loss function at the zero initial point is depthinvariant, which makes training very deep models no more difficult than shallow ones. Shortcuts of higher depth result in an extremely flat (high-order) stationary point initially, from which the optimization algorithm is hard to escape. The 1shortcut, however, is essentially equivalent to no shortcuts. Extensive experiments are provided accompanying our theoretical results. We show that initializing the network to small weights with 2-shortcuts achieves significantly better results than random Gaussian (Xavier) initialization, orthogonal initialization, and shortcuts of deeper depth, from various perspectives ranging from final loss, learning dynamics and stability, to the behavior of the Hessian along the learning process.
1 INTRODUCTION
Residual network (ResNet) was first proposed in He et al. (2015a) and extended in He et al. (2016). It followed a principled approach to add shortcut connections every two layers to a VGG-style network (Simonyan & Zisserman, 2014). The new network becomes easier to train, and achieves both lower training and test errors. Using the new structure, He et al. (2015a) managed to train a network with 1001 layers, which was virtually impossible before. Unlike Highway Network (Srivastava et al., 2015a;b) which not only has shortcut paths but also borrows the idea of gates from LSTM (Sainath et al., 2013), ResNet does not have gates. Later He et al. (2016) found that by keeping a clean shortcut path, residual networks will perform even better.
Many attempts have been made to improve ResNet to a further extent. “ResNet in ResNet” (Targ et al., 2016) adds more convolution layers and data paths to each layer, making it capable of representing several types of residual units. “ResNets of ResNets” (Zhang et al., 2016) construct multilevel shortcut connections, which means there exist shortcuts that skip multiple residual units. Wide Residual Networks (Zagoruyko & Komodakis, 2016) makes the residual network shorter but wider, and achieves state of the art results on several datasets while using a shallower network. Moreover, some existing models are also reported to be improved by shortcut connections, including Inceptionv4 (Szegedy et al., 2016), in which shortcut connections make the deep network easier to train.
Why are residual networks so easy to train? He et al. (2015a) suggests that layers in residual networks are learning residual mappings, making them easier to represent identity mappings, which prevents the networks from degradation when the depths of the networks increase. However, Veit et al. (2016) claims that ResNets are actually ensembles of shallow networks, which means they do not solve the problem of training deep networks completely.
We propose a theoretical explanation for the great performance of ResNet. We concur with He et al. (2015a) that the key contribution of ResNet should be some special structure of the loss function that makes training very deep models no more difficult than shallow ones. Analysis, however, seems non-trivial. Quoting He et al. (2015a):
“But if F has only a single layer, Eqn.(1) is similar to a linear layer: y = W1x + x, for which we have not observed advantages. ”
“Deeper non-bottleneck ResNets (e.g., Fig. 5 left) also gain accuracy from increased depth (as shown on CIFAR-10), but are not as economical as the bottleneck ResNets. So the usage of bottleneck designs is mainly due to practical considerations. We further note that the degradation problem of plain nets is also witnessed for the bottleneck designs. ”
Their empirical observations are inspiring. First, the 1-shortcuts mentioned in the first paragraph do not work. Second, noting that the non-bottleneck ResNets have 2-shortcuts, but the bottleneck ResNets use 3-shortcuts, one sees that shortcuts with depth three also do not work. Hence, a reasonable theoretical explanation must be able to distinguish the 2-shortcut from shortcuts of other depths, and clearly demonstrate why the 2-shortcuts are special and are able to ease the optimization process so significantly for deep models, while shortcuts of other depths may not do the job.
FIXAiming at explaining the performance of 2-shortcuts, we need to eliminate other variables that maycontribute to the success of ResNet. Indeed, one may argue that the deep structure of ResNet may give it better representation power (lower approximation error), which contributes to lower training errors. To eliminate this effect, we focus on deep linear networks, where deeper models do not have better approximation properties. The special role of 2-shortcuts naturally arises in the study.
2 MAIN RESULTS
Our work reveals that non-degenerate depth-invariant initial condition numbers, a unique property of residual networks with 2-shortcuts, contributed to the success of ResNet. In fact, in a linear network that will be defined rigorously later, the condition number of Hessian of the Frobenius loss function at the zero initial point is
cond(H) = √ cond((ΣXX − ΣY X)T (ΣXX − ΣY X)), (1)
which is independent of the number of layers. Here ΣXX and ΣY X denote the input-input and the output-input correlation matrices, defined in Section 3.3. The condition number of a possibly non-PSD matrix is defined as: Definition 1. The condition number of a matrix A is defined as
cond(A) = σmax(A)
σmin(A) , (2)
where σmax(A) and σmin(A) are the maximum and minimum of singular values of A. In particular, if A is normal, i.e. ATA = AAT , the definition can be simplified to 1
cond(A) = |λ(A)|max |λ(A)|min , (3)
where |λ(A)|max and |λ(A)|min are the maximum and minimum of the absolute values of eigenvalues of A.
Moreover, the zero initial point for ResNet with 2-shortcuts is in fact a so-called strict saddle point (Ge et al., 2015), which are proved to be easy to escape from.
Why shortcuts of other depths do not work? We show that the Hessian at the zero initial point for the 1-shortcut ResNet has condition number growing unboundedly for deep nets. As is well known in convex optimization theory, large condition numbers can have enormous adversarial impact on the convergence of first order methods (Nemirovski, 2005). Hence, it is quite clear that starting training at a point with a huge condition number would make the algorithm very difficult to escape from the initial point, making 1-shortcut ResNet no better than conventional approaches.
1The equivalence of Equation (2) and Equation (3) can be proved easily using the eigenvalue decomposition ofA. Note that as Hessians are symmetric (if all the second derivatives are continuous), we will use Equation (3) for their condition numbers. As the |λ|min of Hessian is usually very unstable, we calculated |λ|max|λ|(0.1) to represent condition numbers instead, where |λ|(0.1) is the 10th percentile of the absolute values of eigenvalues. Note that the Hessian at zero initial point for 2-shortcut ResNet also has a nice structure of spectrum: see Theorem 1 for details.
For shortcuts with depth deeper than two, the Hessian at the zero initial point is a zero matrix, making it a higher-order stationary point. Intuitively, the higher order the stationary point is, the harder it is to escape from it. Indeed, it is supported both in theory (Anandkumar & Ge, 2016) and by our experiments.
One may still ask: why are we interested in the Hessian at the zero initial point? It is because in order for the outputs of deep neural networks not explode, the singular values of the mapping of each layer are not supposed to be deviating too much from one. Indeed, it is because it is extremely challenging to keep ∏num of layers i=1 λi from exploding or vanishing without keeping all of the λi having unit norm. However, by design ResNet with shortcuts already have an identity mapping every few layers, which forces the mappings inside the shortcuts to have small operator norms. Hence, analyzing the network at zero initial point gives a decent characterization of the searching environment of the optimization algorithm.
FIXOur results also shows that the form of Hessian is more important than the existance of nonlinearitieswhen training the networks. The behaviors of the networks we studied are consistent across both linear and nonlinear structures, where networks with clearer Hessians are much easier to achieve lower training errors.
On the other hand, our experiments reveal that orthogonal initialization (Saxe et al., 2013) is suboptimal. Although better than Xavier initialization (Glorot & Bengio, 2010), the initial condition numbers of the networks still explode as the networks become deeper, which means the networks are still initialized on “bad” submanifolds that are hard to optimize using gradient descent.
3 MODEL
3.1 DEEP LINEAR NETWORKS
Deep linear networks are feed-forward neural networks that only contain linear units, which means their input-output mappings are simply linear transformations. Apparently, increasing their depths will not affect the representational power of the networks. However, linear networks with depth deeper than one show nonlinear dynamics of training (Saxe et al., 2013). As a result, analyzing the training of deep linear networks gives us a better understanding of the training of non-linear networks.
Much theoretical work has been done on deep linear networks. Kawaguchi (2016) extended the work of Choromanska et al. (2015a;b) and proved that with few assumptions, every local minimum point in deep linear networks is a global minimum point. This means that the difficulties in the training of deep linear networks mostly come from saddle points on the loss surfaces, which are also the main causes of slow learning in nonlinear networks (Pascanu et al., 2014).
Saxe et al. (2013) studied the dynamics of training using gradient descent. They found that for a special class of initial conditions, which could be obtained from greedy layerwise pre-training, the training time for a deep linear network with an infinity depth can still be finite. Furthermore, they found that by setting the initial weights to random orthogonal matrices (produced by performing QR or SVD decompositions on random Gaussian matrices), the network will still have a depth independent learning time. They argue that it is caused by the eigenvalue and singular value spectra of the end-to-end linear transformation. When using orthogonal initialization, the overall transformation is an orthogonal matrix, which has all the singular values equal to 1. In the meantime, when using scaled Gaussian initialization, most of the singular values are close to zero, making the network unsuitable for backpropagating errors. However, this explanation is not sufficient to prove that the training difficulty of orthogonal initialized networks is depth-invariant. It only gives us an intuition on why orthogonal initialization performs better than scaled Gaussian initialization.
Thus, we use deep linear networks to study the effect of shortcut connections. After adding the shortcuts, the overall model is still linear and the global minimum does not change.
3.2 NETWORK STRUCTURE
We first generalize a linear network by adding shortcuts to it to make it a linear residual network. We organize the network into R residual units. The r-th residual unit consists of Lr layers whose
weights are W r,1, . . . ,W r,Lr−1, denoted as the transformation path, as well as a shortcut Sr connecting from the first layer to the last one, denoted as the shortcut path. The input-output mapping can be written as
y = R∏ r=1 ( Lr−1∏ l=1 W r,l + Sr)x = Wx, (4)
where x ∈ Rdx , y ∈ Rdy ,W ∈ Rdy×dx . Here if b ≥ a, ∏b i=aW
i denotes W bW (b−1) · · ·W (a+1)W a, otherwise it denotes an identity mapping. The matrix W represents the combination of all the linear transformations in the network. Note that by setting all the shortcuts to zeros, the network will go back to a ( ∑ r(Lr − 1) + 1)-layer plain linear network.
Instead of analyzing the general form, we concentrate on a special kind of linear residual networks, where all the residual units are the same.
Definition 2. A linear residual network is called an n-shortcut linear network if
1. its layers have the same dimension (so that dx = dy);
2. its shortcuts are identity matrices;
3. its shortcuts have the same depth n.
The input-output mapping for such a network becomes
y = R∏ r=1 ( n∏ l=1 W r,l + Idx)x = Wx, (5)
where W r,l ∈ Rdx×dx .
Then we add some activation functions to the networks. We concentrate on the case where activation functions are on the transformation paths, which is also the case in the latest ResNet (He et al., 2016).
Definition 3. An n-shortcut linear network becomes an n-shortcut network if element-wise activation functions σpre(x), σmid(x), σpost(x) are added at the transformation paths, where on a transformation path, σpre(x) is added before the first weight matrix, σmid(x) is added between two weight matrixes and σpost(x) is added after the last weight matrix.
Note that n-shortcut linear networks are special cases of n-shortcut networks, where all the activation functions are identity mappings.
3.3 OPTIMIZATION
We denote the collection of all the variable weight parameters in an n-shortcut linear network as w. Consider m training samples {xµ, yµ}, µ = 1, . . . ,m. Using Frobenius loss, for an n-shortcut linear network, we define the loss function as follows:
L(w) = 1
2m m∑ µ=1 ‖yµ −Wxµ‖22 = 1 2m ‖Y −WX‖2F , (6)
where xµ, yµ are the µ-th columns of X,Y , and ‖·‖F denotes the Frobenius norm. Using gradient descent with learning rate α, we have the weights updating rules as
∆W r,l = α(W rafterW r,l after) T (ΣY X −WΣXX)(W r,lbeforeW r before) T , (7)
where ΣXX and ΣY X denote the input-input and the output-input correlation matrices, defined as
ΣXX = 1
m m∑ µ=1 xµ(xµ)T (8)
ΣY X = 1
m m∑ µ=1 yµ(xµ)T . (9)
Here W rbefore,W r after denote the linear mappings before and after the r-th residual unit, W r,lbefore,W r,l after denote the linear mappings before and after W
r,l within the transformation path of the r-th residual unit. In other words, the overall transformation can be represented as
y = W rafter(W r,l afterW r,lW r,lbefore + Idx)W r beforex. (10)
4 THEORETICAL STUDY
4.1 INITIAL POINT PROPERTIES
Before we analyze the initial point properties of n-shortcut networks, we have to choose the way to initialize them.
NEW ResNet uses MSRA initialization (He et al., 2015b). It is a kind of scaled Gaussian
initialization that tries to keep the variances of signals along a transformation path, which is also the idea behind Xavier initialization (Glorot & Bengio, 2010). However, because of the shortcut paths, the output variance of the entire network will actually explode as the network becomes deeper. Batch normalization units partly solved this problem in ResNet, but still they cannot prevent the large output variance in a deep network.
NEWA simple idea is to zero initialize all the weights, so that the output variances of residual unitsstay the same along the network. It is worth noting that as found in He et al. (2015a), the deeper ResNet has smaller magnitudes of layer responses. This phenomenon has been confirmed in our experiments. As illustrated in Figure 2 and Figure 3, the deeper a residual network is, the small its average Frobenius norm of weight matrixes is, both during the traning process and when the training ends. Also, Hardt & Ma (2016) proves that if all the weight matrixes have small norms, a linear residual network will have no critical points other than the global optimum.
NEWAll these evidences indicate that zero is spacial in a residual network: as the network becomesdeeper, the training tends to end up around it. Thus, we are looking into the Hessian at zero. As the zero is a saddle point, in our experiments we use zero initialization with small random perturbations to escape from it. We first Xavier initialize the weight matrixes, and then multiply a small constant (0,01) to them.
We begin with the definition of k-th order stationary point.
Definition 4. Suppose function f(x) admits k-th order Taylor expansion at point x0. We say that the point x0 is a k-th order stationary point of f(x) if the corresponding k-th order Taylor expansion of f(x) at x = x0 is a constant: f(x) = f(x0) + o(‖x− x0‖k2).
Now we state our main theorem, whose proof can be found in Appendix A.
Theorem 1. Assume that σmid(0) = σpost(0) = 0 and all of σ (k) pre(0), σ (k) mid(0), σ (k) post(0), 1 ≤ k ≤ max(n− 1, 2) exist. For the loss function of an n-shortcut network, at point zero,
1. if n ≥ 2, it is an (n− 1)th-order stationary point. In particular, if n ≥ 3, the Hessian is a zero matrix;
2. if n = 2, the Hessian can be written as
H =  0 AT A 0 0 AT
A 0 . . .
 , (11)
whose condition number is cond(H) = √ cond((ΣXσpre(X) − ΣY σpre(X))T (ΣXσpre(X) − ΣY σpre(X))), (12)
where A only depends on the training set and the activation functions. Except for degenerate cases, it is a strict saddle point (Ge et al., 2015).
3. if n = 1, the Hessian can be written as
H =  B AT AT · · · AT A B AT · · · AT A A B ...
... ... . . . AT A A · · · A B  (13) where A,B only depend on the training set and the activation functions.
Theorem 1 shows that the condition numbers of 2-shortcut networks are depth-invariant with a nice structure of eigenvalues. Indeed, the eigenvalues of the Hessian H at the zero initial point are multiple copies of ± √ eigs(ATA), and the number of copies is equal to the number of shortcut connections.
The Hessian at zero initial point for the 1-shortcut linear network follows block Toeplitz structure, which has been well studied in the literature. In particular, its condition number tends to explode as the number of layers increase (Gray, 2006).
The assumptions hold for most activation functions including tanh, symmetric sigmoid and ReLU (Nair & Hinton, 2010). Note that although ReLU does not have derivatives at zero, one may do a local polynomial approximation to yield σ(k), 1 ≤ k ≤ max(n− 1, 2). To get intuitive explanations of the theorems, imagine changing parameters in an n-shortcut network. One has to change at least n parameters to make any difference in the loss. So zero is an (n− 1)thorder stationary point. Notice that the higher the order of a stationary point, the more difficult for a first order method to escape from it.
On the other hand, if n = 2, one will have to change two parameters in the same residual unit but different weight matrices to affect the loss, leading to a clear block diagonal Hessian.
4.2 LEARNING DYNAMICS
To understand Equation (7) better, we can take n-shortcut linear networks to two extremes. First, when n = 1, let V r,1 = W r,1 + Idx , r = 1, . . . , R− 1. As Idx is a constant, we have
∆V r,1 = α( R−1∏ r′=r+1 V r ′,1)T (ΣY X − ( R−1∏ r′=1 V r ′,1)ΣXX)( r−1∏ r′=1 V r ′,1)T , (14)
which can be seen as a linear network with identity initialization, a special case of orthogonal initialization, if the original 1-shortcut network is zero initialized.
On the other side, if the number of shortcut connections R = 1, the shortcut will only change the distribution of the output training set from Y to Y − X . These two extremes are illustrated in Figure 4
4.3 LEARNING RESULTS
The optimal weights of an n-shortcut linear network can be easily computed via least squares, which leads to
W = Y XT (XXT )−1 = ΣY X(ΣXX)−1, (15)
and the minimum of the loss function is
Lmin = 1
2m ‖Y − ΣY X(ΣXX)−1X‖2F , (16)
where ‖·‖F denotes the Frobenius norm and (ΣXX)−1 denotes any kind of generalized inverse of ΣXX . So given a training set, we can pre-compute its Lmin and use it to evaluate any n-shortcut linear network.
5 EXPERIMENTS
We compare networks with Xavier initialization (Glorot & Bengio, 2010), networks with orthogonal initialization (Saxe et al., 2013) and 2-shortcut networks with zero initialization. The training dynamics of 1-shortcut networks are similar to that of linear networks with orthogonal initialization in our experiments. Setup details can be found in Appendix B.
5.1 INITIAL POINT
We first compute the initial condition numbers for different kinds of linear networks with different depths.
As can be seen in Figure 5, 2-shortcut linear networks have constant condition numbers as expected. On the other hand, when using Xavier or orthogonal initialization in linear networks, the initial condition numbers will go to infinity as the depths become infinity, making the networks hard to train. This also explains why orthogonal initialization is helpful for a linear network, as its initial condition number grows slower than the Xavier initialization.
5.2 LEARNING DYNAMICS
Having a good beginning does not guarantee an easy trip on the loss surface. In order to depict the loss surfaces encountered from different initial points, we plot the maxima and 10th percentiles (instead of minima, as they are very unstable) of the absolute values of Hessians eigenvalues at different losses.
As shown in Figure 6 and Figure 7, the condition numbers of 2-shortcut networks at different losses are always smaller, especially when the loss is large. Also, notice that the condition numbers roughly evolved to the same value for both orthogonal and 2-shortcut linear networks. This may be explained by the fact that the minimizers, as well as any point near them, have similar condition numbers.
Another observation is the changes of negative eigenvalues ratios. Index (ratio of negative eigenvalues) is an important characteristic of a critical point. Usually for the critical points of a neural network, the larger the loss the larger the index (Dauphin et al., 2014). In our experiments, the index of a 2-shortcut network is always smaller, and drops dramatically at the beginning, as shown in Figure 8, left. This might make the networks tend to stop at low critical points.
This is because the initial point is near a saddle point, thus it tends to go towards negative curvature directions, eliminating some negative eigenvalues at the beginning. This phenomenon matches the observation that the gradient reaches its maximum when the index drops dramatically, as shown in Figure 8, right.
5.3 LEARNING RESULTS
We run different networks for 1000 epochs using different learning rates at log scale, and compare the average final losses of the optimal learning rates.
Figure 9 shows the results for linear networks. Just like their depth-invariant initial condition numbers, the final losses of 2-shortcut linear networks stay close to optimal as the networks become deeper. Higher learning rates can also be applied, resulting in fast learning in deep networks.
Then we add ReLUs to the mid positions of the networks. To make a fair comparison, the numbers of ReLU units in different networks are the same when the depths are the same, so 1-shortcut and 3-shortcut networks are omitted. The result is shown in Figure 10.
Note that because of the nonlinearities, the optimal losses vary for different networks with different depths. It is usually thought that deeper networks can represent more complex models, leading to smaller optimal losses. However, our experiments show that linear networks with Xavier or orthogonal initialization have difficulties finding these optimal points, while 2-shortcut networks find these optimal points easily as they did without nonlinear units.
6 FUTURE DIRECTIONS
Further studies should concentrate on the behavior of shortcut connections on convolution networks, as well as the influences of batch normalization units (Ioffe & Szegedy, 2015) in ResNet. Meanwhile, it would be very interesting to extend the insights obtained in this paper to recurrent neural networks such as LSTM (Sainath et al., 2013).
A PROOFS OF THEOREMS
Definition 5. The elements in Hessian of an n-shortcut network is defined as
Hind(w1),ind(w2) = ∂2L
∂w1∂w2 , (17)
where L is the loss function, and the indices ind(·) is ordered lexicographically following the four indices (r, l, j, i) of the weight variable wr,li,j . In other words, the priority decreases along the index of shortcuts, index of weight matrix inside shortcuts, index of column, and index of row.
Note that the collection of all the weight variables in the n-shortcut network is denoted as w. We study the behavior of the loss function in the vicinity of w = 0.
Lemma 1. Assume that w1 = wr1,l1i1,j1 , · · · , wN = w rN ,lN iN ,jN are N parameters of an n-shortcut network. If ∂ 2L
∂w1···∂wN ∣∣∣ w=0
is nonzero, there exists r and k1, · · · , kn such that rkm = r and lkm = m for m = 1, · · · , n.
Proof. Assume there does not exist such r and k1, · · · , kn, then for all the shortcut units r = 1, · · · , R, there exists a weight matrix l such that none of w1, · · · , wN is in W r,l, so all the transformation paths are zero, which means W = Idx . Then ∂2L ∂w1···∂wN ∣∣∣ w=0
= 0, leading to a contradiction.
Lemma 2. Assume that w1 = wr1,l1i1,j1 , w2 = w r2,l2 i2,j2 , r1 ≤ r2. Let L0(w1, w2) denotes the loss function with all the parameters except w1 and w2 set to 0, w′1 = w 1,l1 i1,j1 , w′2 = w 1+1(r1 6=r2),l2 i2,j2 . Then ∂ 2L0(w1,w2) ∂w1∂w2 |(w1,w2)=0 = ∂2L0(w ′ 1,w ′ 2) ∂w′1∂w ′ 2 |(w′1,w′2)=0.
Proof. As all the residual units expect unit r1 and r2 are identity transformations, reordering residual units while preserving the order of units r1 and r2 will not affect the overall transformation, i.e. L0(w1, w2)|w1=a,w2=b = L′0(w′1, w′2)|w′1=a,w′2=b. So ∂2L0(w1,w2) ∂w1∂w2
|(w1,w2)=0 = ∂2L0(w ′ 1,w ′ 2)
∂w′1∂w ′ 2 |(w′1,w′2)=0.
Proof of Theorem 1. Now we can prove Theorem 1 with the help of the previously established lemmas.
1. Using Lemma 1, for an n-shortcut network, at zero, all the k-th order partial derivatives of the loss function are zero, where k ranges from 1 to n − 1. Hence, the initial point zero is a (n− 1)th-order stationary point of the loss function.
2. Consider the Hessian in n = 2 case. Using Lemma 1 and Lemma 2, the form of Hessian can be directly written as Equation (11), as illustrated in Figure 11.
So we have
eigs(H) = eigs(
[ 0 AT
A 0
] ) = ± √ eigs(ATA). (18)
Thus cond(H) = √
cond(ATA), which is depth-invariant. Note that the dimension of A is d2x × d2x. To get the expression of A, consider two parameters that are in the same residual unit but different weight matrices, i.e. w1 = w r,2 i1,j1 , w2 = w r,1 i2,j2 .
Residual Unit 1
Re sid
ua l U
ni t 1
Residual Unit 2
Re sid
ua l U
ni t 2
…
…
… …
So the eigenvalues of H becomes eigs(H) = ±σ′mid(0)σ′post(0) √
eigs((ΣXσpre(X) − ΣY σpre(X))T (ΣXσpre(X) − ΣY σpre(X))), (22)
which leads to Equation (12).
3. Now consider the Hessian in the n = 1 case. Using Lemma 2, the form of Hessian can be directly written as Equation (13).
To get the expressions of A and B in σpre(x) = σpost(x) = x case, consider two parameters that are in the same residual units, i.e. w1 = w r,1 i1,j1 , w2 = w r,1 i2,j2 .
We have
B(j1−1)dx+i1,(j2−1)dx+i2 = ∂2L
∂w1∂w2
∣∣∣ w=0
(23)
=
{ 1 m ∑m µ=1 x µ j1 xµj2 i1 = i2
0 i1 6= i2 (24)
Rearrange the order of variables using P , we have
B = P Σ XX
. . . ΣXX PT . (25) Then consider two parameters that are in different residual units, i.e. w1 = w r1,1 i1,j1 , w2 = wr2,1i2,j2 , r1 > r2.
We have
A(j1−1)dx+i1,(j2−1)dx+i2 = ∂2L
∂w1∂w2
∣∣∣ w=0
(26)
=  1 m ∑m µ=1(x µ i1 − yµi1)x µ j2 + xµj1x µ j2 j1 = i2, i1 = i2 1 m ∑m µ=1(x µ i1 − yµi1)x µ j2 j1 = i2, i1 6= i2 1 m ∑m µ=1 x µ j1 xµj2 j1 6= i2, i1 = i2
0 j1 6= i2, i1 6= i2
(27)
In the same way, we can rewrite A as
A = Σ XX − ΣY X
. . . ΣXX − ΣY X
PT +B. (28)
B EXPERIMENT SETUP
We took the experiments on whitened versions of MNIST. 10 greatest principal components are kept for the dataset inputs. The dataset outputs are represented using one-hot encoding. The network was trained using gradient descent. For every epoch, the Hessians of the networks were calculated using the method proposed in (Bishop, 1992). As the |λ|min of Hessian is usually very unstable, we calculated |λ|max|λ|(0.1) to represent condition number instead, where |λ|(0.1) is the 10
th percentile of the absolute values of eigenvalues.
As pre, mid or post positions are not defined in linear networks without shortcuts, when comparing Xavier or orthogonal initialized linear networks to 2-shortcut networks, we added ReLUs at the same positions in linear networks as in 2-shortcuts networks.
This paper studies the optimization issue of linear ResNet, and shows mathematically that for 2-shortcuts and zero initialization, the Hessian has condition number independent of depth. I skimmed through the proof but have not checked them carefully. 
This result is a nice observation for training deep linear networks.  But I do not think the paper has fully resolved the linear vs nonlinear issue. Some question:
1. Though the revision has added some results using ReLU units, it seems it is only added to the mid positions of the network (sec 5.3), is this how it is typically done in ResNet? Moreover, ReLU is not differentiable at zero point, which does not satisfy the condition you had in Theorem 1. Why not use differentiable activations like sigmoid or tanh?
2. From equation (22) in the appendix, it seems for nonlinear activations, the condition number depends on the derivative \sigma^\prime at 0. Therefore, if we use tanh which has derivative 1 at zero, the condition number is the same for linear and tanh activations. But this probably is not enough to explain the bit difference in performance or optimization for linear and nonlinear networks, or how the situations evolve after learning the 0 point.
3. As for the success of ResNet (or convnets in general) in computer vision, I believe there are more types of nonlinearity such as pooling? Can the result here generalizes to pooling as well?
Minor: 
- sec 1 last paragraph, low approximation error typically means more powerful model class and better training error, but not necessarily better test error
- sec 4.1 what do you mean by "zero initialization with small random perturbations"? why not exactly zero initialization, how large is the random perturbation?