An emerging design principle in deep learning is that each layer of a deep artificial neural network should be able to easily express the identity transformation. This idea not only motivated various normalization techniques, such as batch normalization, but was also key to the immense success of residual networks. In this work, we put the principle of identity parameterization on a more solid theoretical footing alongside further empirical progress. We first give a strikingly simple proof that arbitrarily deep linear residual networks have no spurious local optima. The same result for feed-forward networks in their standard parameterization is substantially more delicate. Second, we show that residual networks with ReLu activations have universal finite-sample expressivity in the sense that the network can represent any function of its sample provided that the model has more parameters than the sample size. Directly inspired by our theory, we experiment with a radically simple residual architecture consisting of only residual convolutional layers and ReLu activations, but no batch normalization, dropout, or max pool. Our model improves significantly on previous all-convolutional networks on the CIFAR10, CIFAR100, and ImageNet classification benchmarks.
An emerging design principle in deep learning is that each layer of a deep artificial neural network should be able to easily express the identity transformation. This idea not only motivated various normalization techniques, such as batch normalization, but was also key to the immense success of residual networks. In this work, we put the principle of identity parameterization on a more solid theoretical footing alongside further empirical progress. We first give a strikingly simple proof that arbitrarily deep linear residual networks have no spurious local optima. The same result for feed-forward networks in their standard parameterization is substantially more delicate. Second, we show that residual networks with ReLu activations have universal finite-sample expressivity in the sense that the network can represent any function of its sample provided that the model has more parameters than the sample size. Directly inspired by our theory, we experiment with a radically simple residual architecture consisting of only residual convolutional layers and ReLu activations, but no batch normalization, dropout, or max pool. Our model improves significantly on previous all-convolutional networks on the CIFAR10, CIFAR100, and ImageNet classification benchmarks.
1 INTRODUCTION
Traditional convolutional neural networks for image classification, such as AlexNet (Krizhevsky et al. (2012)), are parameterized in such a way that when all trainable weights are 0, a convolutional layer represents the 0-mapping. Moreover, the weights are initialized symmetrically around 0. This standard parameterization makes it non-trivial for a convolutional layer trained with stochastic gradient methods to preserve features that were already good. Put differently, such convolutional layers cannot easily converge to the identity transformation at training time.
This shortcoming was observed and partially addressed by Ioffe & Szegedy (2015) through batch normalization, i.e., layer-wise whitening of the input with a learned mean and covariance. But the idea remained somewhat implicit until residual networks (He et al. (2015); He et al. (2016)) explicitly introduced a reparameterization of the convolutional layers such that when all trainable weights are 0, the layer represents the identity function. Formally, for an input x, each residual layer has the form x + h(x), rather than h(x). This simple reparameterization allows for much deeper architectures largely avoiding the problem of vanishing (or exploding) gradients. Residual networks, and subsequent architectures that use the same parameterization, have since then consistently achieved state-of-the-art results on various computer vision benchmarks such as CIFAR10 and ImageNet.
1.1 OUR CONTRIBUTIONS
In this work, we consider identity parameterizations from a theoretical perspective, while translating some of our theoretical insight back into experiments. Loosely speaking, our first result underlines how identity parameterizations make optimization easier, while our second result shows the same is true for representation.
Linear residual networks. Since general non-linear neural networks, are beyond the reach of current theoretical methods in optimization, we consider the case of deep linear networks as a simplified model. A linear network represents an arbitrary linear map as a sequence of matrices A` · · ·A2A1. The objective function is E‖y − A` · · ·A1x‖2, where y = Rx for some unknown linear transformation R and x is drawn from a distribution. Such linear networks have been studied actively in recent years as a stepping stone toward the general non-linear case (see Section 1.2). Even though A` · · ·A1 is just a linear map, the optimization problem over the factored variables (A`, . . . , A1) is non-convex.
In analogy with residual networks, we will instead parameterize the objective function as
min A1,...,A`
E‖y − (I +A`) · · · (I +A1)x‖2 . (1.1)
To give some intuition, when the depth ` is large enough, we can hope that the target function R has a factored representation in which each matrix Ai has small norm. Any symmetric positive semidefinite matrix O can, for example, be written as a product O = O` · · ·O1, where each Oi = O1/` is very close to the identity for large ` so that Ai = Oi − I has small spectral norm. We first prove that an analogous claim is true for all linear transformations R. Specifically, we prove that for every linear transformation R, there exists a global optimizer (A1, . . . , A`) of (1.1) such that for large enough depth `,
max 1≤i≤`
‖Ai‖ ≤ O(1/`). (1.2)
Here, ‖A‖ denotes the spectral norm of A. The constant factor depends on the conditioning of R. We give the formal statement in Theorem 2.1. The theorem has the interesting consequence that as the depth increases, smaller norm solutions exist and hence regularization may offset the increase in parameters.
Having established the existence of small norm solutions, our main result on linear residual networks shows that the objective function (1.1) is, in fact, easy to optimize when all matrices have sufficiently small norm. More formally, letting A = (A1, . . . , A`) and f(A) denote the objective function in (1.1), we can show that the gradients of vanish only when f(A) = 0 provided that maxi ‖Ai‖ ≤ O(1/`). See Theorem 2.2. This result implies that linear residual networks have no critical points other than the global optimum. In contrast, for standard linear neural networks we only know, by work of Kawaguchi (2016) that these networks don’t have local optima except the global optimum, but it doesn’t rule out other critical points. In fact, setting Ai = 0 will always lead to a bad critical point in the standard parameterization.
Universal finite sample expressivity. Going back to non-linear residual networks with ReLU activations, we can ask: How expressive are deep neural networks that are solely based on residual layers with ReLU activations? To answer this question, we give a very simple construction showing that such residual networks have perfect finite sample expressivity. In other words, a residual network with ReLU activations can easily express any functions of a sample of size n, provided that it has sufficiently more than n parameters. Note that this requirement is easily met in practice. On CIFAR 10 (n = 50000), for example, successful residual networks often have more than 106 parameters. More formally, for a data set of size nwith r classes, our construction requiresO(n log n+r2) parameters. Theorem 3.2 gives the formal statement.
Each residual layer in our construction is of the form x + V ReLU(Ux), where U and V are linear transformations. These layers are significantly simpler than standard residual layers, which typically have two ReLU activations as well as two instances of batch normalization.
The power of all-convolutional residual networks. Directly inspired by the simplicity of our expressivity result, we experiment with a very similar architecture on the CIFAR10, CIFAR100, and ImageNet data sets. Our architecture is merely a chain of convolutional residual layers each with a single ReLU activation, but without batch normalization, dropout, or max pooling as are common in standard architectures. The last layer is a fixed random projection that is not trained. In line with our theory, the convolutional weights are initialized near 0, using Gaussian noise mainly as a symmetry breaker. The only regularizer is standard weight decay (`2-regularization) and there is no need for dropout. Despite its simplicity, our architecture reaches 6.38% top-1 classification error on the CIFAR10 benchmark (with standard data augmentation). This is competitive with the best
residual network reported in He et al. (2015), which achieved 6.43%. Moreover, it improves upon the performance of the previous best all-convolutional network, 7.25%, achieved by Springenberg et al. (2014). Unlike ours, this previous all-convolutional architecture additionally required dropout and a non-standard preprocessing (ZCA) of the entire data set. Our architecture also improves significantly upon Springenberg et al. (2014) on both Cifar100 and ImageNet.
1.2 RELATED WORK
Since the advent of residual networks (He et al. (2015); He et al. (2016)), most state-of-the-art networks for image classification have adopted a residual parameterization of the convolutional layers. Further impressive improvements were reported by Huang et al. (2016) with a variant of residual networks, called dense nets. Rather than adding the original input to the output of a convolutional layer, these networks preserve the original features directly by concatenation. In doing so, dense nets are also able to easily encode an identity embedding in a higher-dimensional space. It would be interesting to see if our theoretical results also apply to this variant of residual networks.
There has been recent progress on understanding the optimization landscape of neural networks, though a comprehensive answer remains elusive. Experiments in Goodfellow et al. (2014) and Dauphin et al. (2014) suggest that the training objectives have a limited number of bad local minima with large function values. Work by Choromanska et al. (2015) draws an analogy between the optimization landscape of neural nets and that of the spin glass model in physics (Auffinger et al. (2013)). Soudry & Carmon (2016) showed that 2-layer neural networks have no bad differentiable local minima, but they didn’t prove that a good differentiable local minimum does exist. Baldi & Hornik (1989) and Kawaguchi (2016) show that linear neural networks have no bad local minima. In contrast, we show that the optimization landscape of deep linear residual networks has no bad critical point, which is a stronger and more desirable property. Our proof is also notably simpler illustrating the power of re-parametrization for optimization. Our results also indicate that deeper networks may have more desirable optimization landscapes compared with shallower ones.
2 OPTIMIZATION LANDSCAPE OF LINEAR RESIDUAL NETWORKS
Consider the problem of learning a linear transformation R : Rd → Rd from noisy measurements y = Rx+ ξ, where ξ ∈ N (0, Idd) is a d-dimensional spherical Gaussian vector. Denoting by D the distribution of the input data x, let Σ = Ex∼D[xx>] be its covariance matrix.
There are, of course, many ways to solve this classical problem, but our goal is to gain insights into the optimization landscape of neural nets, and in particular, residual networks. We therefore parameterize our learned model by a sequence of weight matrices A1, . . . , A` ∈ Rd×d,
h0 = x , hj = hj−1 +Ajhj−1 , ŷ = h` . (2.1)
Here h1, . . . , h`−1 are the ` − 1 hidden layers and ŷ = h` are the predictions of the learned model on input x. More succinctly, we have
ŷ = (Idd +A`) . . . (Id +A1)x .
It is easy to see that this model can express any linear transformationR.We will useA as a shorthand for all of the weight matrices, that is, the ` × d × d-dimensional tensor the contains A1, . . . , A` as slices. Our objective function is the maximum likelihood estimator,
f(A, (x, y)) = ‖ŷ − y‖2 = ‖(Id +A`) . . . (Id +A1)x−Rx− ξ‖2 . (2.2) We will analyze the landscape of the population risk, defined as,
f(A) := E [f(A, (x, y))] .
Recall that ‖Ai‖ is the spectral norm ofAi. We define the norm |||·||| for the tensorA as the maximum of the spectral norms of its slices,
|||A||| := max 1≤i≤` ‖Ai‖ .
The first theorem of this section states that the objective function f has an optimal solution with small |||·|||-norm, which is inversely proportional to the number of layers `. Thus, when
the architecture is deep, we can shoot for fairly small norm solutions. We define γ := max{| log σmax(R)|, | log σmin(R)|}. Here σmin(·), σmax(·) denote the least and largest singular values of R respectively. Theorem 2.1. Suppose ` ≥ 3γ. Then, there exists a global optimum solution A? of the population risk f(·) with norm
|||A?||| ≤ 2( √ π + √ 3γ)2/` .
Here γ should be thought of as a constant since if R is too large (or too small), we can scale the data properly so that σmin(R) ≤ 1 ≤ σmax(R). Concretely, if σmax(R)/σmin(R) = κ, then we can scaling for the outputs properly so that σmin(R) = 1/ √ κ and σmax(R) = √ κ. In this case, we
have γ = log √ κ, which will remain a small constant for fairly large condition number κ. We also point out that we made no attempt to optimize the constant factors here in the analysis. The proof of Theorem 2.1 is rather involved and is deferred to Section A.
Given the observation of Theorem 2.1, we restrict our attention to analyzing the landscape of f(·) in the set of A with |||·|||-norm less than τ ,
Bτ = {A ∈ R`×d×d : |||A||| ≤ τ} .
Here using Theorem 2.1, the radius τ should be thought of as on the order of 1/`. Our main theorem in this section claims that there is no bad critical point in the domain Bτ for any τ < 1. Recall that a critical point has vanishing gradient. Theorem 2.2. For any τ < 1, we have that any critical point A of the objective function f(·) inside the domain Bτ must also be a global minimum.
Theorem 2.2 suggests that it is sufficient for the optimizer to converge to critical points of the population risk, since all the critical points are also global minima.
Moreover, in addition to Theorem 2.2, we also have that any A inside the domain Bτ satisfies that
‖∇f(A)‖2F ≥ 4`(1− τ) `−1σmin(Σ) 2(f(A)− Copt) . (2.3)
Here Copt is the global minimal value of f(·) and ‖∇f(A)‖F denotes the euclidean norm1 of the `× d× d-dimensional tensor∇f(A). Note that σmin(Σ) denote the minimum singular value of Σ. Equation (2.3) says that the gradient has fairly large norm compared to the error, which guarantees convergence of the gradient descent to a global minimum (Karimi et al. (2016)) if the iterates stay inside the domain Bτ , which is not guaranteed by Theorem 2.2 by itself. Towards proving Theorem 2.2, we start off with a simple claim that simplifies the population risk. We also use ‖·‖F to denote the Frobenius norm of a matrix. Claim 2.3. In the setting of this section, we have,
f(A) = ∥∥∥((Id +A`) . . . (Id +A1)−R)Σ1/2∥∥∥2
F + C . (2.4)
Here C is a constant that doesn’t depend on A, and Σ1/2 denote the square root of Σ, that is, the unique symmetric matrix B that satisfies B2 = Σ.
Proof of Claim 2.3. Let tr(A) denotes the trace of the matrixA. LetE = (Id+A`) . . . (Id+A1)−R. Recalling the definition of f(A) and using equation (2.2), we have
f(A) = E [ ‖Ex− ξ‖2 ] (by equation (2.2))
= E [ ‖Ex‖2 + ‖ξ‖2 − 2〈Ex, ξ〉 ] = E [ tr(Exx>E>) ] + E [ ‖ξ‖2 ] (since E [〈Ex, ξ〉] = E [〈Ex,E [ξ|x]〉] = 0)
= tr ( E E [ xx> ] E> ) + C (where C = E[xx>])
= tr(EΣE>) + C = ‖EΣ1/2‖2F + C . (since E [ xx> ] = Σ)
1That is, ‖T‖F := √∑ ijk T 2 ijk.
Next we compute the gradients of the objective function f(·) from straightforward matrix calculus. We defer the full proof to Section A. Lemma 2.4. The gradients of f(·) can be written as,
∂f
∂Ai = 2(Id +A>` ) . . . (Id +A > i+1)EΣ(Id +A > i−1) . . . (Id +A > 1 ) , (2.5)
where E = (Id +A`) . . . (Id +A1)−R .
Now we are ready to prove Theorem 2.2. The key observation is that each matric Aj has small norm and cannot cancel the identity matrix. Therefore, the gradients in equation (2.5) is a product of non-zero matrices, except for the error matrix E. Therefore, if the gradient vanishes, then the only possibility is that the matrix E vanishes, which in turns implies A is an optimal solution.
Proof of Theorem 2.2. Using Lemma 2.4, we have,∥∥∥∥ ∂f∂Ai ∥∥∥∥ F = 2 ∥∥(Id +A>` ) . . . (Id +A>i+1)EΣ(Id +A>i−1) . . . (Id +A>1 )∥∥F (by Lemma 2.4)
≥ 2 ∏ j 6=i σmin(Id +A>i ) · σmin(Σ)‖E‖F (by Claim C.2)
≥ 2(1− τ)`−1σmin(Σ)‖E‖ . (since σmin(Id +A) ≥ 1− ‖A‖) It follows that
‖∇f(A)‖2F = ∑̀ i=1 ∥∥∥∥ ∂f∂Ai ∥∥∥∥2 F ≥ 4`(1− τ)`−1σmin(Σ)2‖E‖2
≥ 4`(1− τ)`−1σmin(Σ)2(f(A)− C) (by the definition of E and Claim 2.3) ≥ 4`(1− τ)`−1σmin(Σ)2(f(A)− Copt) .
(since Copt = minA f(A) ≥ C by Claim 2.3)
Therefore we complete the proof of equation (2.3). Finally, ifA is a critical point, namely,∇f(A) = 0, then by equation (2.3) we have that f(A) = Copt. That is, A is a global minimum.
3 REPRESENTATIONAL POWER OF THE RESIDUAL NETWORKS
In this section we characterize the finite-sample expressivity of residual networks. We consider a residual layers with a single ReLU activation and no batch normalization. The basic residual building block is a function TU,V,s(·) : Rk → Rk that is parameterized by two weight matrices U ∈ R×k, V ∈ Rk×k and a bias vector s ∈ Rk,
TU,V,s(h) = V ReLu(Uh+ s) . (3.1) A residual network is composed of a sequence of such residual blocks. In comparison with the full pre-activation architecture in He et al. (2016), we remove two batch normalization layers and one ReLU layer in each building block.
We assume the data has r labels, encoded as r standard basis vectors in Rr, denoted by e1, . . . , er. We have n training examples (x(1), y(1)), . . . , (x(n), y(n)), where x(i) ∈ Rd denotes the i-th data and y(i) ∈ {e1, . . . , er} denotes the i-th label. Without loss of generality we assume the data are normalized so that x(i) = 1. We also make the mild assumption that no two data points are very close to each other. Assumption 3.1. We assume that for every 1 ≤ i < j ≤ n, we have ‖x(i) − x(j)‖2 ≥ ρ for some absolute constant ρ > 0.
Images, for example, can always be imperceptibly perturbed in pixel space so as to satisfy this assumption for a small but constant ρ.
Under this mild assumption, we prove that residual networks have the power to express any possible labeling of the data as long as the number of parameters is a logarithmic factor larger than n.
Theorem 3.2. Suppose the training examples satisfy Assumption 3.1. Then, there exists a residual network N (specified below) with O(n log n + r2) parameters that perfectly expresses the training data, i.e., for all i ∈ {1, . . . , n}, the network N maps x(i) to y(i).
It is common in practice that n > r2, as is for example the case for the Imagenet data set where n > 106 and r = 1000.
We construct the following residual net using the building blocks of the form TU,V,s as defined in equation (3.1). The network consists of `+ 1 hidden layers h0, . . . , h`, and the output is denoted by ŷ ∈ Rr. The first layer of weights matricesA0 maps the d-dimensional input to a k-dimensional hidden variable h0. Then we apply ` layers of building block T with weight matrices Aj , Bj ∈ Rk×k. Finally, we apply another layer to map the hidden variable h` to the label ŷ in Rk. Mathematically, we have
h0 = A0x ,
hj = hj−1 + TAj ,Bj ,bj (hj−1), ∀j ∈ {1, . . . , `} ŷ = h` + TA`+1,B`+1,s`+1(h`) .
We note that here A`+1 ∈ Rk×r and B`+1 ∈ Rr×r so that the dimension is compatible. We assume the number of labels r and the input dimension d are both smaller than n, which is safely true in practical applications.2 The hyperparameter k will be chosen to be O(log n) and the number of layers is chosen to be ` = dn/ke. Thus, the first layer has dk parameters, and each of the middle ` building blocks contains 2k2 parameters and the final building block has kr+r2 parameters. Hence, the total number of parameters is O(kd+ `k2 + rk + r2) = O(n log n+ r2).
Towards constructing the network N of the form above that fits the data, we first take a random matrix A0 ∈ Rk×d that maps all the data points x(i) to vectors h(i)0 := A0x(i). Here we will use h (i) j to denote the j-th layer of hidden variable of the i-th example. By Johnson-Lindenstrauss Theorem (Johnson & Lindenstrauss (1984), or see Wikipedia (2016)), with good probability, the resulting vectors h(i)0 ’s remain to satisfy Assumption 3.1 (with slightly different scaling and larger constant ρ), that is, any two vectors h(i)0 and h (j) 0 are not very correlated.
Then we construct ` middle layers that maps h(i)0 to h (i) ` for every i ∈ {1, . . . , n}. These vectors h (i) ` will clustered into r groups according to the labels, though they are in the Rk instead of in Rr as desired. Concretely, we design this cluster centers by picking r random unit vectors q1, . . . , qr in Rk. We view them as the surrogate label vectors in dimension k (note that k is potentially much smaller than r). In high dimensions (technically, if k > 4 log r) random unit vectors q1, . . . , qr are pair-wise uncorrelated with inner product less than < 0.5. We associate the i-th example with the target surrogate label vector v(i) defined as follows,
if y(i) = ej , then v(i) = qj . (3.2)
Then we will construct the matrices (A1, B1), . . . , (A`, B`) such that the first ` layers of the network maps vector h(i)0 to the surrogate label vector v
(i). Mathematically, we will construct (A1, B1), . . . , (A`, B`) such that
∀i ∈ {1, . . . , n}, h(i)` = v (i) . (3.3)
Finally we will construct the last layer TA`+1,B`+1,b`+1 so that it maps the vectors q1, . . . , qr ∈ Rk to e1, . . . , er ∈ Rr,
∀j ∈ {1, . . . , r}, qj + TA`+1,B`+1,b`+1(qj) = ej . (3.4) Putting these together, we have that by the definition (3.2) and equation (3.3), for every i, if the label is y(i) is ej , then h (i) ` will be qj . Then by equation (3.4), we have that ŷ (i) = qj + TA`+1,B`+1,b`+1(qj) = ej . Hence we obtain that ŷ(i) = y(i).
The key part of this plan is the construction of the middle ` layers of weight matrices so that h(i)` = v(i). We encapsulate this into the following informal lemma. The formal statement and the full proof is deferred to Section B.
2In computer vision, typically r is less than 103 and d is less than 105 while n is larger than 106
Lemma 3.3 (Informal version of Lemma B.2). In the setting above, for (almost) arbitrary vectors h(1)0 , . . . , h (n) 0 and v
(1), . . . , v(n) ∈ {q1, . . . , qr}, there exists weights matrices (A1, B1), . . . , (A`, B`), such that,
∀i ∈ {1, . . . , n}, h(i)` = v (i) .
We briefly sketch the proof of the Lemma to provide intuitions, and defer the full proof to Section B. The operation that each residual block applies to the hidden variable can be abstractly written as,
ĥ→ h+ TU,V,s(h) . (3.5)
where h corresponds to the hidden variable before the block and ĥ corresponds to that after. We claim that for an (almost) arbitrary sequence of vectors h(1), . . . , h(n), there exist TU,V,s(·) such that operation (3.5) transforms k vectors of h(i)’s to an arbitrary set of other k vectors that we can freely choose, and maintain the value of the rest of n − k vectors. Concretely, for any subset S of size k, and any desired vector v(i)(i ∈ S), there exist U, V, s such that
v(i) = h(i) + TU,V,s(h(i)) ∀i ∈ S h(i) = h(i) + TU,V,s(h(i)) ∀i 6∈ S (3.6)
This claim is formalized in Lemma B.1. We can use it repeatedly to construct ` layers of building blocks, each of which transforms a subset of k vectors in {h(1)0 , . . . , h (n) 0 } to the corresponding vectors in {v(1), . . . , v(n)}, and maintains the values of the others. Recall that we have ` = dn/ke layers and therefore after ` layers, all the vectors h(i)0 ’s are transformed to v
(i)’s, which complete the proof sketch.
4 POWER OF ALL-CONVOLUTIONAL RESIDUAL NETWORKS
Inspired by our theory, we experimented with all-convolutional residual networks on standard image classification benchmarks.
4.1 CIFAR10 AND CIFAR100
Our architectures for CIFAR10 and CIFAR100 are identical except for the final dimension corresponding to the number of classes 10 and 100, respectively. In Table 1, we outline our architecture. Each residual block has the form x+C2(ReLU(C1x)), where C1, C2 are convolutions of the specified dimension (kernel width, kernel height, number of input channels, number of output channels). The second convolution in each block always has stride 1, while the first may have stride 2 where indicated. In cases where transformation is not dimensionality-preserving, the original input x is adjusted using averaging pooling and padding as is standard in residual layers.
We trained our models with the Tensorflow framework, using a momentum optimizer with momentum 0.9, and batch size is 128. All convolutional weights are trained with weight decay 0.0001. The initial learning rate is 0.05, which drops by a factor 10 and 30000 and 50000 steps. The model reaches peak performance at around 50k steps, which takes about 24h on a single NVIDIA Tesla K40 GPU. Our code can be easily derived from an open source implementation3 by removing batch normalization, adjusting the residual components and model architecture. An important departure from the code is that we initialize a residual convolutional layer of kernel size k × k and c output channels using a random normal initializer of standard deviation σ = 1/k2c, rather than 1/k √ c used for standard convolutional layers. This substantially smaller weight initialization helped training, while not affecting representation.
A notable difference from standard models is that the last layer is not trained, but simply a fixed random projection. On the one hand, this slightly improved test error (perhaps due to a regularizing effect). On the other hand, it means that the only trainable weights in our model are those of the convolutions, making our architecture “all-convolutional”.
3https://github.com/tensorflow/models/tree/master/resnet
An interesting aspect of our model is that despite its massive size of 13.59 million trainable parameters, the model does not seem to overfit too quickly even though the data set size is 50000. In contrast, we found it difficult to train a model with batch normalization of this size without significant overfitting on CIFAR10.
Table 2 summarizes the top-1 classification error of our models compared with a non-exhaustive list of previous works, restricted to the best previous all-convolutional result by Springenberg et al. (2014), the first residual results He et al. (2015), and state-of-the-art results on CIFAR by Huang et al. (2016). All results are with standard data augmentation.
4.2 IMAGENET
The ImageNet ILSVRC 2012 data set has 1, 281, 167 data points with 1000 classes. Each image is resized to 224 × 224 pixels with 3 channels. We experimented with an all-convolutional variant of the 34-layer network in He et al. (2015). The original model achieved 25.03% classification error. Our derived model has 35.7M trainable parameters. We trained the model with a momentum optimizer (with momentum 0.9) and a learning rate schedule that decays by a factor of 0.94 every two epochs, starting from the initial learning rate 0.1. Training was distributed across 6 machines
updating asynchronously. Each machine was equipped with 8 GPUs (NVIDIA Tesla K40) and used batch size 256 split across the 8 GPUs so that each GPU updated with batches of size 32.
In contrast to the situation with CIFAR10 and CIFAR100, on ImageNet our all-convolutional model performed significantly worse than its original counterpart. Specifically, we experienced a significant amount of underfitting suggesting that a larger model would likely perform better.
Despite this issue, our model still reached 35.29% top-1 classification error on the test set (50000 data points), and 14.17% top-5 test error after 700, 000 steps (about one week of training). While no longer state-of-the-art, this performance is significantly better than the 40.7% reported by Krizhevsky et al. (2012), as well as the best all-convolutional architecture by Springenberg et al. (2014). We believe it is quite likely that a better learning rate schedule and hyperparameter settings of our model could substantially improve on the preliminary performance reported here.
5 CONCLUSION
Our theory underlines the importance of identity parameterizations when training deep artificial neural networks. An outstanding open problem is to extend our optimization result to the non-linear case where each residual has a single ReLU activiation as in our expressivity result. We conjecture that a result analogous to Theorem 2.2 is true for the general non-linear case. Unlike with the standard parameterization, we see no fundamental obstacle for such a result.
We hope our theory and experiments together help simplify the state of deep learning by aiming to explain its success with a few fundamental principles, rather than a multitude of tricks that need to be delicately combined. We believe that much of the advances in image recognition can be achieved with residual convolutional layers and ReLU activations alone. This could lead to extremely simple (albeit deep) architectures that match the state-of-the-art on all image classification benchmarks.
A MISSING PROOFS IN SECTION 2
In this section, we give the complete proofs for Theorem 2.1 and Lemma 2.4, which are omitted in Section 2.
A.1 PROOF OF THEOREM 2.1
It turns out the proof will be significantly easier if R is assumed to be a symmetric positive semidefinite (PSD) matrix, or if we allow the variables to be complex matrices. Here we first give a proof sketch for the first special case. The readers can skip it and jumps to the full proof below. We will also prove stronger results, namely, |||A?||| ≤ 3γ/`, for the special case. When R is PSD, it can be diagonalized by orthonormal matrix U in the sense that R = UZU>, where Z = diag(z1, . . . , zd) is a diagonal matrix with non-negative diagonal entries z1, . . . , zd. Let A?1 = · · · = A?` = U diag(z 1/` i )U > − Id, then we have
(Id +A?` ) · · · (Id +A?1) = (U diag(z 1/` i )U >)` = U diag(z 1/` i ) `U (since U>U = Id)
= UZU> = R .
We see that the network defined by A? reconstruct the transformation R, and therefore it’s a global minimum of the population risk (formally see Claim 2.3 below). Next, we verify that each of the A?j has small spectral norm:
‖A?j‖ = ‖Id− U diag(z 1/` i )U >)‖ = ‖U(Id− diag(zi)1/`)U>‖ = ‖Id− diag(zi)1/`‖ (since U is orthonormal)
= max i |z1/`i − 1| . (A.1)
Since σmin(R) ≤ zi ≤ σmax(R), we have ` ≥ 3γ ≥ | log zi|. It follows that
|z1/`i − 1| = |e (log zi)/` − 1| ≤ 3|(log zi)/`| ≤ 3γ/` . (since |ex − 1| ≤ 3|x| for all |x| ≤ 1)
Then using equation (A.1) and the equation above, we have that |||A||| ≤ maxj‖A?j‖ ≤ 3γ/`, which completes the proof for the special case.
Next we give the formal full proof of Theorem 2.1.
Proof of Theorem 2.1. We assume the dimension d is an even number. The odd case has very similar proof and is left to the readers. LetR = UKV > be its singular value decomposition, where U ,V are two orthonormal matrices andK is a diagonal matrix. Since U is a normal matrix (that is, U satisfies thatUU> = U>U ), by Claim C.1, we have thatU can be block-diagnolaized by orthonormal matrix S into U = SDS−1, where D = diag(D1, . . . , Dd/2) is a real block diagonal matrix with each block Di being of size 2× 2. SinceU is orthonormal, U has all its eigenvalues lying on the unit circle (in complex plane). SinceD and U are unitarily similar to each other, D also has eigenvalues lying on the unit circle, and so does each of the block Di. This means that each Di is a 2× 2 dimensional rotation matrix. Each rotation matrix can be written as T (θ) = [ cos θ − sin θ sin θ cos θ ] . Suppose Di = T (θi) where θi ∈ [−π, π]. Then
we have that Di = T (θi/q)q for any integer q (that is chosen later). Let W = diag(T (θi/q)). Therefore, it follows that D = diag(Di) = W q. Moreover, we have U = SDS−1 = (SWS−1)q . Therefore, let B1 = B2 = · · · = Bq = Id − SWS−1, then we have U = (Id + Bq) . . . (Id + B1). We verify the spectral norm of these matrices are indeed small,
‖Bj‖ = ∥∥Id− SWS−1∥∥ = ∥∥S(Id−W )S−1∥∥
= ‖Id−W‖ (since S is unitary) = max i∈[d/2] ‖T (0)− T (θi/q)‖ (since W = diag(T (θi/q)) is block diagonal)
= max | sin(θi/q)| ≤ π/q .
Similarly, we can choose B′1, . . . , B ′ q with ‖Cj‖ ≤ π/q so that V > = (Id +B′q) . . . (Id +B′1).
Last, we deal with the diagonal matrix K. Let K = diag(ki). We have min ki = σmin(R),max ki = σmax(R). Then, we can write K = (K ′)p where K ′ = diag(k 1/p i ) and p is an integer to be chosen later. We have that ‖K ′ − Id‖ ≤ max |k1/pi − 1| ≤ max |elog ki·1/p − 1|. When p ≥ γ = max{log max ki,− log min ki} = max{log σmax(R),− log σmin(R)}, we have that
‖K ′ − Id‖ ≤ max |elog ki·1/p − 1| ≤ 3 max | log ki · 1/p| = 3γ/p . (since |ex − 1| ≤ 3|x| for |x| ≤ 1)
Let B′′1 = · · · = B′′p = K ′ − Id and then we have K = (Id + B′′p ) · · · (Id + B′′1 ). Finally, we choose p = ` √ 3γ
2( √ π+ √ 3γ)
and q = ` √ π√
π+ √ 3γ , 4and let A2p+q = Bq, · · · = Ap+q+1 = B1, Ap+q = B′′p , . . . , Aq+1 = B ′′ 1 , Aq = B ′ q, . . . , A1 = B ′ 1. We have that 2q + ` = 1 and
R = UKV > = (Id +A`) . . . (Id +A1) .
Moreover, we have |||A||| ≤ max{‖Bj‖, ‖B′j‖.‖B′′j ‖} ≤ π/q + 3γ/p ≤ 2( √ π + √ 3γ)2/`, as desired.
4Here for notational convenience, p, q are not chosen to be integers. But rounding them to closest integer will change final bound of the norm by small constant factor.
A.2 PROOF OF LEMMA 2.4
We compute the partial gradients by definition. Let ∆j ∈ Rd×d be an infinitesimal change to Aj . Using Claim 2.3, consider the Taylor expansion of f(A1, . . . , A` + ∆j , . . . , A`)
f(A1, . . . , A` + ∆j , . . . , A`) = ∥∥∥((Id +A`) · · · (Id +Aj + ∆j) . . . (Id +A1)−R)Σ1/2∥∥∥2
F = ∥∥∥((Id +A`) · · · (Id +A1)−R)Σ1/2 + (Id +A`) · · ·∆j . . . (Id +A1)Σ1/2∥∥∥2
F = ∥∥∥(Id +A`) · · · (Id +A1)−R)Σ1/2∥∥∥2
F +
2〈((Id +A`) · · · (Id +A1)−R)Σ1/2, (Id +A`) · · ·∆j . . . (Id +A1)Σ1/2〉+O(‖∆j‖2F ) = f(A) + 2〈(Id +A>` ) . . . (Id +A>j+1)EΣ(Id +A>j−1) . . . (Id +A>1 ),∆j〉+O(‖∆j‖2F ) .
By definition, this means that the ∂f∂Aj = 2(Id + A > ` ) . . . (Id + A > j+1)EΣ(Id + A > j−1) . . . (Id + A>1 ).
B MISSING PROOFS IN SECTION 3
In this section, we provide the full proof of Theorem 3.2. We start with the following Lemma that constructs a building block T that transform k vectors of an arbitrary sequence of n vectors to any arbitrary set of vectors, and main the value of the others. For better abstraction we use α(i),β(i) to denote the sequence of vectors.
Lemma B.1. Let S ⊂ [n] be of size k. Suppose α(1), . . . , α(n) is a sequences of n vectors satisfying a) for every 1 ≤ i ≤ n, we have 1−ρ′ ≤ ‖αi‖2 ≤ 1+ρ′, and b) if i 6= j and S contains at least one of i, j, then ‖α(i)−β(j)‖ ≥ 3ρ′. Let β(1), . . . , β(n) be an arbitrary sequence of vectors. Then, there exists U, V ∈ Rk×k, s such that for every i ∈ S, we have TU,V,s(α(i)) = β(i) − α(i), and moreover, for every i ∈ [n]\S we have TU,V,s(α(i)) = 0.
We can see that the conclusion implies
β(i) = α(i) + TU,V,s(α(i)) ∀i ∈ S α(i) = α(i) + TU,V,s(α(i)) ∀i 6∈ S
which is a different way of writing equation (3.6).
Proof of Lemma B.1. Without loss of generality, suppose S = {1, . . . , k}. We construct U, V, s as follows. Let the i-th row of U be α(i) for i ∈ [k], and let s = −(1− 2ρ′) · 1 where 1 denotes the all 1’s vector. Let the i-column of V be 1‖α(i)‖2−(1−2ρ′) (β (i) − α(i)) for i ∈ [k].
Next we verify that the correctness of the construction. We first consider 1 ≤ i ≤ k. We have that Uα(i) is a a vector with i-th coordinate equal to ‖α(i)‖2 ≥ 1 − ρ′. The j-th coordinate of Uα(i) is equal to 〈α(j), α(i)〉, which can be upperbounded using the assumption of the Lemma by
〈α(j), α(i)〉 = 1 2
( ‖α(i)‖2 + ‖α(j)‖2 ) − ‖α(i) − α(j)‖2 ≤ 1 + ρ′ − 3ρ′ ≤ 1− 2ρ′ . (B.1)
Therefore, this means Uα(i) − (1 − 2ρ′) · 1contains a single positive entry (with value at least ‖α(i)‖2 − (1 − 2ρ′) ≥ ρ′), and all other entries being non-positive. This means that ReLu(Uα(i) +b) = ( ‖α(i)‖2 − (1− 2ρ′) ) ei where ei is the i-th natural basis vector. It follows that V ReLu(Uα(i) + b) = (‖α(i)‖2 − (1− 2ρ′))V ei = β(i) − α(i).
Finally, consider n ≥ i > k. Then similarly to the computation in equation (B.1), Uα(i) is a vector with all coordinates less than 1− 2ρ′. Therefore Uα(i) + b is a vector with negative entries. Hence we have ReLu(Uα(i) + b) = 0, which implies V ReLu(Uα(i) + b) = 0.
Now we are ready to state the formal version of Lemma 3.3.
Lemma B.2. Suppose a sequence of n vectors z(1), . . . , z(n) satisfies a relaxed version of Assumption 3.1: a) for every i, 1− ρ′ ≤ ‖z(i)‖2 ≤ 1 + ρ′ b) for every i 6= j, we have ‖z(i) − z(j)‖2 ≥ ρ′;. Let v(1), . . . , v(n) be defined above. Then there exists weigh matrices (A1, B1), . . . , (A`, B`), such that given ∀i, h(i)0 = z(i), we have,
∀i ∈ {1, . . . , n}, h(i)` = v (i) .
We will use Lemma B.1 repeatedly to construct building blocks TAj ,Bk,sj (·), and thus prove Lemma B.2. Each building block TAj ,Bk,sj (·) takes a subset of k vectors among {z(1), . . . , z(n)} and convert them to v(i)’s, while maintaining all other vectors as fixed. Since they are totally n/k layers, we finally maps all the z(i)’s to the target vectors v(i)’s.
Proof of Lemma B.2. We use Lemma B.1 repeatedly. Let S1 = [1, . . . , k]. Then using Lemma B.1 with α(i) = z(i) and β(i) = v(i) for i ∈ [n], we obtain that there exists A1, B1, b1 such that for i ≤ k, it holds that h(i)1 = z(i) + TA1,B1,b1(z(i)) = v(i), and for i ≥ k, it holds that h (i) 1 = z(i) + TA1,B1,b1(z(i)) = z(i). Now we construct the other layers inductively. We will construct the layers such that the hidden variable at layer j satisfies h(i)j = v
(i) for every 1 ≤ i ≤ jk, and h(i)j = z(i) for every n ≥ i > jk. Assume that we have constructed the first j layer and next we use Lemma B.1 to construct the j + 1 layer. Then we argue that the choice of α(1) = v(1), . . . , α(jk) = v(jk), α(jk+1) = z(jk+1), . . . , α(n) = z(n), and S = {jk + 1, . . . , (j + 1)k} satisfies the assumption of Lemma B.1. Indeed, because qi’s are chosen uniformly randomly, we have w.h.p for every s and i, 〈qs, z(i)〉 ≤ 1− ρ′. Thus, since v(i) ∈ {q1, . . . , qr}, we have that v(i) also doesn’t correlate with any of the z(i). Then we apply Lemma B.1 and conclude that there exists Aj+1 = U,Bj+1 = V, bj+1 = s such that TAj+1,bj+1,bj+1(v(i)) = 0 for i ≤ jk, TAj+1,bj+1,bj+1(z(i)) = v(i)− z(i) for jk < i ≤ (j + 1)k, and TAj+1,bj+1,bj+1(z(i)) = 0 for n ≥ i > (j + 1)k. These imply that
h (i) j+1 = h (i) j + TAj+1,bj+1,bj+1(v (i)) = v(i) ∀1 ≤ i ≤ jk
h (i) j+1 = h (i) j + TAj+1,bj+1,bj+1(z (i)) = v(i) ∀jk + 1 ≤ i ≤ (j + 1)k
h (i) j+1 = h (i) j + TAj+1,bj+1,bj+1(z (i)) = z(i) ∀(j + 1)k < i ≤ n Therefore we constructed the j + 1 layers that meets the inductive hypothesis for layer j + 1. Therefore, by induction we get all the layers, and the last layer satisfies that h(i)` = v
(i) for every example i.
Now we ready to prove Theorem 3.2, following the general plan sketched in Section 3.
Proof of Theorem 3.2. We use formalize the intuition discussed below Theorem 3.2. First, take k = c(log n)/ρ2 for sufficiently large absolute constant c (for example, c = 10 works), by JohnsonLindenstrauss Theorem (Johnson & Lindenstrauss (1984), or see Wikipedia (2016)) we have that when A0 is a random matrix with standard normal entires, with high probability, all the pairwise distance between the the set of vectors {0, x(1), . . . , x(n)} are preserved up to 1 ± ρ/3 factor. That is, we have that for every i, 1−ρ/3 ≤ ‖A0x(i)‖ ≤ 1+ρ/3, and for every i 6= j, ‖A0x(i)−A0x(j)‖ ≥ ρ(1 − ρ/3) ≥ 2ρ/3. Let z(i) = A0x(i) and ρ′ = ρ/3. Then we have z(i)’s satisfy the condition of Lemam B.2. We pick r random vectors q1, . . . , qr in Rk. Let v(1), . . . , v(n) be defined as in equation (3.2). Then by Lemma B.2, we can construct matrices (A1, B1), . . . , (A`, B`) such that
h (i) ` = v (i) . (B.2)
Note that v(i) ∈ {q1, . . . , qr}, and qi’s are random unit vector. Therefore, the choice of α(1) = q1, . . . , α
(r) = qr, β(1) = e1, . . . , β(r) = er, and satisfies the condition of Lemma B.1, and using Lemma B.1 we conclude that there exists A`+1, B`+1, s`+1 such that
ej = vj + TA`+1,B`+1,b`+1(vj), for every j ∈ {1, . . . , r} .. (B.3)
By the definition of v(i) in equation (3.2) and equation (B.2), we conclude that ŷ(i) = h(i)` + TA`+1,B`+1,b`+1(h (i) ` ) = y (i)., which complete the proof.
C TOOLBOX
In this section, we state two folklore linear algebra statements. The following Claim should be known, but we can’t find it in the literature. We provide the proof here for completeness. Claim C.1. Let U ∈ Rd×d be a real normal matrix (that is, it satisfies UU> = U>U ). Then, there exists an orthonormal matrix S ∈ Rd×d such that
U = SDS> ,
where D is a real block diagonal matrix that consists of blocks with size at most 2× 2. Moreover, if d is even, then D consists of blocks with size exactly 2× 2.
Proof. Since U is a normal matrix, it is unitarily diagonalizable (see Weisstein (2016) for backgrounds). Therefore, there exists unitary matrix V in Cd×d and diagonal matrix in Cd×d such that U has eigen-decomposition U = V ΛV ∗. Since U itself is a real matrix, we have that the eigenvalues (the diagonal entries of Λ) come as conjugate pairs, and so do the eigenvectors (which are the columns of V ). That is, we can group the columns of V into pairs (v1, v̄1), . . . , (vs, v̄s), vs+1, . . . , vt, and let the corresponding eigenvalues be λ1, λ̄1, . . . , λλs , λ̄s, λs+1, . . . , λt. Here λs+1, . . . , λt ∈ R. Then we get that U =∑s i=1 2<(viλiv∗i ) + ∑t i=s+1 viλiv > i . Let Qi = <(viλiv∗i ), then we have that Qi is a real matrix of rank-2. Let Si ∈ Rd×2 be a orthonormal basis of the column span ofQi and then we have thatQi can be written as Qi = SiDiS>i where Di is a 2× 2 matrix. Finally, let S = [S1, . . . , Ss, vs+1, . . . , vt], and D = diag(D1, . . . , Ds, λs+1, . . . , λt) we complete the proof.
The following Claim is used in the proof of Theorem 2.2. We provide a proof here for completeness. Claim C.2 (folklore). For any two matrices A,B ∈ Rd×d, we have that
‖AB‖F ≥ σmin(A)‖B‖F .
Proof. Since σmin(A)2 is the smallest eigenvalue of A>A, we have that
B>A>AB B> · σmin(A)2Id ·B .
Therefore, it follows that
‖AB‖2F = tr(B>A>AB) ≥ tr(B> · σmin(A)2Id ·B) = σmin(A) 2tr(B>B) = σmin(A)2‖B‖2F .
Taking square root of both sides completes the proof.

Paper Summary:
Authors investigate identity re-parametrization in the linear and the non linear case.
Detailed comments:
— Linear Residual Network:
The paper shows that for a linear residual network any critical point is a global optimum. This problem is non convex it is interesting that this simple re-parametrization leads to such a result.
 — Non linear Residual Network:
Authors propose a construction that maps the points to their labels via a resnet , using an initial random projection, followed by a residual block that clusters the data based on their label, and a last layer that maps the clusters to the label.
1- In Eq 3.4  seems the dimensions are not matching q_j in R^k and e_j in R^r. please clarify
2- The construction seems fine, but what is special about the resnet here in this construction? One can do a similar construction if we did not have the identity? can you discuss this point?
In the linear case it is clear from a spectral point of view how the identity is helping the optimization. Please provide some intuition.
3-   Existence of a network in the residual  class that overfits does it give us any intuition on why residual network outperform other architectures? What does an existence result of such a network tell us about its representation power ?
A simple linear model under the assumption that points can not be too close can overfit the data, and get fast convergence rate (see for instance tsybakov noise condition).
4- What does the construction tell us about the number of layers?
5- clustering the activation independently from the label, is an old way to pretrain the network. One could use those centroids as weights for the next layer (this is also related to Nystrom approximation see for instance


Classifying question sentences into their corresponding categories is an important task with wide applications, for example in many websites’ FAQ sections. However, traditional question classification techniques do not fully utilize the wellprepared answer data which has great potential for improving question representation and could lead to better classification performance. In order to encode answer information into question representation, we first introduce novel group sparse autoencoders which could utilize the group information in the answer set to refine question representation. We then propose a new group sparse convolutional neural network which could naturally learn the question representation with respect to their corresponding answers by implanting the group sparse autoencoders into the traditional convolutional neural network. The proposed model show significant improvements over strong baselines on four datasets.
1 INTRODUCTION
Question classification has applications in question answering (QA), dialog systems, etc., and has been increasingly popular in recent years. Most existing approaches to this problem simply use existing sentence modeling frameworks and treat questions as general sentences, without any special treatment. For example, several recent efforts employ Convolutional Neural Networks (CNNs) to achieve remarkably strong performance in the TREC question classification task as well as other sentence classification tasks such as sentiment analysis (Kim, 2014; Kalchbrenner et al., 2014; Ma et al., 2015).
We argue, however, that the general sentence modeling frameworks neglect several unique properties in question classification not found in other sentence classification tasks (such as sentimental classification or sarcasm detection), which we detail below:
• The categories for most sentence classification tasks are flat and coarse (notable exceptions such as the Reuters Corpus RCV1 (Lewis et al., 2004) notwithstanding), and in many cases, even binary (i.e. sarcasm detection). However, question sentences commonly belong to multiple categories, and these categories often have a hierarchical (tree or DAG) structure such as those from the New York State DMV FAQ section 1 in Fig. 1.
• Question sentences from different categories often share similar information or language patterns. This phenomenon becomes more obvious when categories are hierarchical. Fig. 2 shows one example of questions sharing similar information from different categories. This cross-category shared patterns are not only shown in questions but can also be found in answers corresponding to these questions.
• Another unique characteristic for question classification is the well prepared answer set with detailed descriptions or instructions for each corresponding question category. These answer sets generally cover a broader range of vocabulary (than the questions themselves) and carry more distinctive semantic meanings for each class. We believe there is great
1 http://nysdmv.custhelp.com/app/home
potential to enhance the representation of questions with extra information from corresponding answer sets.
To exploit the hierarchical and overlapping structures in question categories and extra information from answer sets, we consider dictionary learning (Aharon et al., 2005; Roth & Black, 2005; Lee et al., 2007; Candè & Wakin, 2008; Kreutz-Delgado et al., 2003; Rubinstein et al., 2010) which is one common approach for representing samples from a vast, correlated groups with external information. This learning procedure first builds a dictionary with a series of grouped bases. These bases can be initialized randomly or from external data (from the answer set in our case) and optimized during training through Sparse Group Lasso (SGL) (Simon et al., 2013). There are many promising improvements which have been achieved recently by this grouped-dictionary learning-based methods (Zhao et al., 2016; Rao et al., 2016). We also showcase some preliminary experiments in Section 6 for question classification with SGL, and the performance is indeed extraordinary compared with baselines but still lose to the CNNs-based method. Considering the unique advantages from the SGL-based model and the CNNs-based model, we believe that performance of question classification will have another boost if we could put SGL-based and CNNs-based model within the same end-to-end framework. This requires us to design a new neural-based model which behaves similarly with SGL.
Based on the above observations, we first propose a novel Group Sparse Autoencoders (GSA). The objective of GSA and SGL are very similar. The encoding matrix of GSA (like the dictionary in SGL) is grouped into different categories. The bases in different groups can be either initialized randomly or by the sentences in corresponding answer categories. Each question sentence will be reconstructed by some bases within a few groups. To the best of our knowledge, GSA is the first full neural network based model with group sparse constraints. GSA has can be either linear or nonlinear encoding or decoding while SGL is restrained to be linear. In order to incorporate both advantages from GSA and CNNs, we then propose a new Group Sparse Convolutional Neural Networks (GSCNNs) by implanting the GSA into CNNs between the convolutional layer and the classification layer. GSCNNs are jointly trained end-to-end neural-based framework for getting question representations with group sparse constraint from both answer and question sets. Experiments show significant improvements over strong baselines on four datasets.
2 PRELIMINARIES: SPARSE AUTOENCODERS
We first review the basic autoencoders and sparse autoencoders to establish the mathematical notations. Then we propose our new autoencoder with group sparse constraints in later section.
2.1 BASIC AUTOENCODERS
As introduced in (Bengio et al., 2007), autoencoder is an unsupervised neural network which could learn the hidden representations of input samples. An autoencoder takes an input instance z ∈ Rd, and then maps it into a hidden space in the form of h ∈ Rs through a deterministic mapping function h = Φθ(z) = Φ(Wz + b), where θ = {W, b}. W is a d × s projection matrix and b is the bias term. The projection function can be linear or non-linear function such as sigmoid. This projection process often can be recognized as encoding process. The encoded hidden representation is then mapped back to the original input space to reconstruct a vector ẑ ∈ Rd with function ẑ = Φθ′(h) = Φ(W ′h + c) with θ′ = {W ′, c}. The reverse projection matrix W ′ may optionally be constrained by W ′ = WT . This reverse operation can be recognized as a decoding process which tries to reconstruct a new vector z such that the difference between ẑ and z are as small as possible by minimizing the average reconstruction error:
J(W, b, c) = argmin W,b,c
1
n n∑ i=1 L(z(i), ẑ(i)) = argmin W,b,c 1 n n∑ i=1 L ( z(i),ΦWT ,c(ΦW,b(z (i))) )
(1)
where L is a loss function such as minimum square error L(z, ẑ) = ‖z− ẑ‖2. Depending on the applications, this loss function also can be defined in form of computing the reconstruction cross-entropy between z and ẑ:
LC(z, ẑ) = − d∑
k=1
(zk log ẑk + (1− zk) log(1− ẑk))
When the dimensionality of the hidden space s is smaller than the dimensionality of the input space d. The network is forced to learn a compressed representation of the input. If there is structure or feature correlation in the data, the linear autoencoders often ends up learning a low-dimensional representation like PCA. Most of the time, autoencoders learns a compressed representation when the number of hidden units s being small. However, when the number of hidden units becomes larger than the dimensionality of input space, there are still some interesting structure that can be discovered by imposing other constraints on the network. The following discussed sparse constraints is one of them.
2.2 SPARSE AUTOENCODERS
Sparse autoencoders (Ng, 2011; Makhzani & Frey, 2014) shows interesting results of getting visualization of the hidden layers. Recall that hij represents the activations of j
th hidden unit for a given specific input zi. Then the average activation of hidden unit j (average over the training batch) can be defined as:
ρ̂j = 1
m m∑ i=1 hij (2)
wherem is the number of samples in training batch. The goal of sparse autoencoders is to enforce the constraint:
ρ̂j = ρ (3)
where ρ is the sparsity parameter which controls how sparse you want the hidden representation to be. Typically ρ is set to be a small value close to zero. In order to satisfy this constraint, the activations of hidden layer must mostly be close to 0.
In order to achieve the above objective, there will be an extra penalty term in our optimization function which tries to reconstruct the original input with as few hidden layer activations as possible. The most commonly used penalty term (Ng, 2011) is as follows:
s∑ j=1 KL(ρ||ρ̂j) = s∑ j=1 ρ log ρ ρ̂j + (1− ρ) log 1− ρ 1− ρ̂j (4)
where s is the number of units in hidden layer, and j is the index of the hidden unit. This penalty term is based on KL divergence which measures the difference between two different distributions.
Then our new objective of the sparse autoencoders is defined as follows:
Jsparse(W, b, c) = J(W, b, c) + α s∑ j=1 KL(ρ||ρ̂j) (5)
where J(W, b, c) is defined in Eq. 1, and α controls the weights of the sparsity penalty term. Note that the term ρ̂j is implicitly controlled by W , b and c. This is one of the difference between sparse autoencoders and sparse coding which will be discussed in details in Section 6.
3 GROUP SPARSE AUTOENCODERS
As described above, sparse autoencoder has similar objective with sparse coding which tries to find sparse representations for input samples. Inspired by the motivations from group sparse lasso (Yuan & Lin, 2006) and sparse group lasso (Simon et al., 2013), we propose a novel Group Sparse Autoencoders (GSA)in this paper.
Different from sparse autoencoders, in our GSA, the weight matrix is categorized into different groups. For a given input, GSA reconstructs the input signal with the activations from only a few groups. Similar to the average activation defined in Eq. 2 for sparse autoencoders, in GSA, we define each grouped average activation for the hidden layer as follows:
η̂p = 1
mg m∑ i=1 g∑ l=1 ||hip,l||2 (6)
where g represents the number of samples in each group, and m represents the number of samples in training batch. η̂j first sums up all the activations within pth group, then computes the average pth group respond across different samples’ hidden activations.
Similar with Eq.4, we also use KL divergence to measure the difference between estimated intra-group activation and goal group sparsity as follows:
G∑ p=1 KL(η||η̂p) = η log η η̂p + (1− η) log 1− η 1− η̂p (7)
whereG is the number of groups. When we only need inter-group constraints, the loss function of autoencoders can be defined as follows:
Jgs(W, b, c) = J(W, b, c) + β g∑ l=1 KL(η||η̂p) (8)
In some certain cases, inter- and intra- group sparsity are preferred and the same time. Then objective can be defined as follows:
Jgs(W, b, c) = J(W, b, c) + α s∑ j=1 KL(ρ||ρ̂j)
+ β G∑ p=1 KL(η||η̂p) (9)
Inter-group sparse autoencoders defined in Eq. 8 has similar functionality with group sparse lasso in (Yuan & Lin, 2006). Inter- and intra- group sparse autoencoders which defined in Eq. 9 behaves similarly to sparse group lasso in (Simon et al., 2013). Different from the sparse coding approach, the encoding and decoding process could be nonlinear while sparse coding is always linear.
Similar to sparse coding approach, the projection matrix in GSA works like a dictionary which includes all the necessary bases for reconstructing the input signal with the activations in the hidden layer. Different initialization methods for projection matrix are described in Section 5.
3.1 VISUALIZATION FOR GROUP SPARSE AUTOENCODERS
In order to have a better understanding of how the GSA behaves, We use MNIST dataset for visualizing the internal parameters of GSA. We visualize the projection matrix in Fig. 3 and the corresponding hidden activation in Fig. 4.
In our experiments, we use 10, 000 samples for training. We set the size of hidden layer as 500 with 10 different groups for GSA. We set the intra-group sparsity ρ equal to 0.3 and inter-group sparsity η equal to 0.2. α and β are equal to 1. On the other hand, we also train the same 10, 000 examples on basic autoencoders with random noise added to the input signal (denoising autoencoders (Vincent et al., 2008)) for better hidden information extraction. We add the same 30% random noise into both models. Note that the group size of this experiments does not have to be set to 10. Since this is the image dataset with digit numbers, we may use fewer groups to train GSA.
In Fig. 3(b), we could find similar patterns within each group. For example, the 8th group in Fig. 3(b) has different forms of digit 0, and 9th group includes different forms of digit 7. However, it is difficult to tell any meaningful patterns from the projection matrix of basic autoencoders in Fig. 3(c).
Fig. 4 shows the hidden activations respect to the input image in Fig. 3(a). From the results, we can tell that most of the activations of hidden layer are in group 1, 2, 6 and 8. And the 8th group has the most significant activations. When we refer this activations to the projection matrix visualization in Fig. 3(b). These results are reasonable since the 8th row has the most similar patterns of digit 0.
GSA could be directly applied to small image data (i.e. MINIST dataset) for pre-training. However, in the tasks which prefer dense, semantic representation (i.e. sentence classification), we still need CNNs to learn the
sentence representation automatically. In this scenario, in order to incorporate both advantages from GSA and CNNs, we propose Group Sparse Convolutional Neural Networks in the following section.
4 GROUP SPARSE CONVOLUTIONAL NEURAL NETS
Convolutional neural networks (CNNs) were first proposed by (LeCun et al., 1995) in computer vision. For a given image, CNNs apply convolution kernels on a series of continuous areas on images. This concept was first adapted to NLP by (Collobert et al., 2011). Recently, many CNNs-based techniques achieve great successes in sentence modeling and classification (Kim, 2014; Kalchbrenner et al., 2014; Ma et al., 2015). For simplicity, we use the sequential CNNs (Kim, 2014) as our baseline.
Following sequential CNNs, one dimensional convolution operates the convolution kernel in sequential order in Eq. 10, where xi ∈ Re represents the e dimensional word representation for the i-th word in the sentence, and ⊕ is the concatenation operator. Therefore xi,j refers to concatenated word vector from the i-th word to the (i+ j)-th word in sentence:
xi,j = xi ⊕ xi+1 ⊕ · · · ⊕ xi+j (10)
A convolution operates a filter w ∈ Rn×e to a window of n words xi,i+n with bias term b′ described in Eq. 11 to produce a new feature.
ai = σ(w · xi,i+n + b′) (11)
where σ is a non-linear activation function such as rectified linear unit (ReLu) or sigmoid function. The filter w is applied to each word in the sentence, generating the feature map a ∈ RL:
a = [a1, a2, · · · , aL] (12)
where L is the length of the sentence.
The convolution described in Eq. 11 can be regarded as feature detection: more similar patterns will return higher activation. In sequential CNNs, max-over-time pooling (Collobert et al., 2011; Kim, 2014) operates over the feature map to get the maximum activation â = max{a} representing the entire feature map. The idea is to detect the strongest activation over time. This pooling strategy also naturally deals with sentence length variations.
In order to capture different aspects of patterns, CNNs usually randomly initialize a set of filters with different sizes and values. Each filter will generate a feature as described above. To take all the features generated by N different filters into count, we use z = [â1, · · · , âN ] as the final representation.
In conventional CNNs, z will be directly fed into classifiers after the sentence representation is obtained, e.g. fully connected neural networks in (Kim, 2014). There is no easy way for CNNs to explore the possible hidden representations with interesting underlaying structures.
In order to obtains the hidden representations for each sentence representation, we proposed a Group Sparse Convolutional Neural Networks (GSCNNs) by placing one extra layer between convolutional layer and classification layer. This extra layer is trying to mimic the functionality of GSA that we introduced in Section 2.
Our proposed framework is shown in Fig. 5. The convolutional layer show in Fig. 5 follows the traditional convolution process which is described previously. After the convolutional layer, we get z which is the feature map for each sentence. The feature maps z is treated as the feature representation for each sentence. In stead of directly feeding z into a fully connected neural network for classification, we enforce the group sparse constraint on z like the group sparse constraint we have on h in Eq. 9. Then, we use the hidden representation h in Eq. 9 as new sentence representation. The last step is feeding the hidden representation h into fully connected neural network for classification. The parameters W , b, and c in Eq. 9 will also be fine tunned during the last step.
In order to improve the robustness of the hidden representation and prevent it from simply learning the identity, we follow the idea of decisioning autoencoders (Vincent et al., 2008) to add random noise (10% in our experiments) into z. The training process of our model is similar to the training process in stack autoencoders (Bengio et al., 2007).
In order to prevent the co-adaptation of the hidden unites, we employ random dropout on penultimate layer (Hinton et al., 2014). We set the drop out rate as 0.5 and learning rate as 0.95 by default. In our experiments, training is done through stochastic gradient descent over shuffled mini-batches with the Adadelta update rule (Zeiler, 2012). All the settings of the CNNs are the same as the settings in (Kim, 2014).
5 EXPERIMENTS
Since there has been little effort to use answer sets in question classification, we did not find any well-fitted datasets which are publicly available. We collected two datasets and use other two well-known datasets in our experiments. The statistics of these datasets is summarized in Table 1. The descriptions of each dataset are as follows:
• TREC The TREC dataset2 is a factoid question classification dataset. The task is to classify each question into one of the 6 different question types (Li & Roth, 2002). The reason we include this factoid questions dataset is to show the effectiveness of the proposed method in an frequently used dataset even there is no categorized answer sets available.
• Insurance This is a private dataset which we collected from a car insurance company’s website. Each question is classified into the 319 possible classes with corresponding answer data. All questions which belongs to the same category share the same answers. All answers are generated manually. Most questions have multiple assigned labels.
• DMV dataset We collected this dataset from New York State DMV’s FAQ website. We will make this data publicly available in the future.
• Yahoo Ans The Yahoo! Answers dataset (Fleming et al., 2012; Shah & Pomerantz, 2010) is a publicly available dataset.3 There are more than 4 million questions with answers. For simplicity reasons, we only randomly sample 8,871 questions from the complete dataset. There are 27 top level categories across different domains. To make our task more realistic and challenging, we test the proposed model with respect to the subcategories and there are 678 classes.
2http://cogcomp.cs.illinois.edu/Data/QA/QC/ 3http://webscope.sandbox.yahoo.com/catalog.php?datatype=l
We only compare our model’s performance with CNNs for two following reasons: we consider our “group sparse” as a modification to the general CNNs for grouped feature selection. This idea is “orthogonal” to any other CNNs-based models and can be easily applied to them; another reason is, as discussed in Sec. 1, we did not find any other model which can be used for comparison in soloving question classification task with answer sets.
The datasets we use in the experiments require the label information for both questions and answers. Besides that, similar with websites’ FAQ section, all the questions which belong to the same category share the same answer sets. Among the above the four datasets, only the Insurance and DMV datasets are well-fitted for our model. The questions which fall into the same category have different answers in Yahoo dataset.
Different ways of initializing the projection matrix in Eq. 9 can be summarized as the followings:
• Random Initialization: when there is no answer corpus available, we first random initialize N vectors (usually N s) to represent the representation from answer set. Then we cluster these N vectors intoG categories with g centroids for each category. These centroids from different categories will be the initialized bases for projection matrixW . This projection matrix will be optimized during training.
• Initialization from Questions: instead of using random initialized vectors, we could also use question sentences for initializing the projection matrix when answer set is not available. We need to pre-train the sentence with CNNs to get the sentence representation. We then select top G largest categories in terms of number of question sentences. Then we get g centroids from each category by k-means. We concatenate these G × g vectors group after group to form the projection matrix. We need to pre-train the sentence with CNNs to get the sentence representation.
• Initialization from Answers: This is the most ideal case. We follow the same procedure from above. The only difference is that we then treat the answer sentence as question sentence to pre-train the CNNs to get answer sentence representation.
Note that projection matrix will be updated during training for better classification performance.
In the cases of single-label classification tasks (TREC and Yahoo dataset), we set the last layer as softmax-layer which tries to get one unique peaky choice across all other labels. But in the cases for multi-label classification (Insurance and DMV dataset), we replace the softmax-layer in CNNs with sigmoid-layer since sigmoid layer predicts each category independently while softmax function has an exclusive property which allows cross influence between categories.
All the experiments results are summarized in Table 2. TREC dataset is factoid question type classification. We include this experiments to show our performance on a frequently used dataset. Proposed method improves marginally over baseline because the sentences are too short in TREC dataset. For Insurance and DMV dataset, the improvement is significant.
In the experiments with Yahoo dataset, the improvement is not as signification as Insurance and DMV. One reason for this is the questions in Yahoo dataset are usually too short, sometime only have 2 to 3 words. When the sentences become shorter, the group information become harder to encode. Another reason is that the questions in Yahoo dataset are always single labeled, and can not fully utilize the benefits of group sparse properties. Yahoo-top shows the results of top categories classification results. We map the subcategories back to the top categories and get the results in Table 2.
Besides the conventional classification tasks, we also test our proposed model on unseen-label experiments. In this experiments, there are a few sub-category labels that are not included in training process. However, we still hope that our model could correctly classify these unseen sub-category label into correct parent category based on the model’s sub-category estimation. In the testing set of Yahoo dataset, we randomly add 100 questions
whose labels are unseen in training set. The classification results of Yahoo-unseen in Table 2 are obtained by mapping the subcategory classification results to top level category and check whether the true label’s top category match with predicted label’s parent category. The improvements are remarkable due to the group information encoding.
6 DISCUSSION
The idea of reforming signal to a sparse representation is first introduced in the domain of compressed sensing (Candè & Wakin, 2008) which achieves great success in signal compression, visualization and classification task. Especially when dictionary is well trained, the performance usually improves significantly, as shown in (Wang et al., 2010; Yang et al., 2009) for image classification tasks. In Table 3, we test the influence of Sparse Group Lasso (SGL) (Simon et al., 2013) with two baseline methods, k-Nearest Neighbor (k-NN) and SVM on the Insurance dataset. We use TF-IDF as feature representation for each question and answer sentence. We first select all the answer sentences from top 20 largest category and then find 10 centroids for each of these categories by k-Means. Then we have a dictionary with 200 centroids with 20 groups. We notice there is a great improvement of performance after we preprocess the original sentence representations with SGL before we use SVM. We further test the performance of CNNs on the same dataset, and CNNs outperforms SVM and k-NN even with SGL because of the well trained sentence representation through CNNs. However, for vanilla CNNs, it is not straightforward to embed SGL into the network and still get good representation for sentences since SGL will break the training error in backpropagation.
However, GSA is fully neural network based framework. Our proposed GSA has similar functionalities to SGL (Yuan & Lin, 2006; Simon et al., 2013), as it is shown in Fig. 3 and Fig. 4, but in different approach. Compared with sparse coding approaches which have intense optimizations on both dictionary and coding, GSA’s optimization is based on simple backpropagation. GSA also can be easily placed into any neural network for joint training. Another advantage of GSA over sparse coding is that the projection function Φ in GSA can be linear or non-linear, while sparse coding always learns linear codings.
7 CONCLUSIONS AND FUTURE WORK
In this paper, we first present a novel GSA framework which functions as dictionary learning and sparse coding models with inter- and intra- group sparse constraints. We also prove GSA’s learning ability by visualizing the projection matrix and activations. We further propose a group sparse convolutional neural networks by embedding GSA into CNNs. We show that CNNs can benefit from GSA by learning more meaningful representation from dictionary.

The paper proposes the group sparse autoencoder that enforces sparsity of the hidden representation group-wise, where the group is formed based on labels (i.e., supervision). The p-th group hidden representation is used for reconstruction with group sparsity penalty, allowing learning more discriminative, class-specific patterns in the dataset. The paper also propose to combine both group-level and individual level sparsity as in Equation (9).
Clarity of the paper is a bit low.
- Do you use only p-th group's activation for reconstruction? If it is true, then for Equation (9) do you use all individual hidden representation for reconstruction or still using the subset of representation corresponding to that class only?
- In Equation (7), RHS misses the summation over p, and wondering it is a simple typo.
- Is the algorithm end-to-end trainable? It seems to me that the group sparse CNN is no more than the GSA whose input data is the feature extracted from sequential CNNs (or any other pretrained CNNs).
Other comments are as follows:
- Furthermore the group sparse autoencoder is (semi-) supervised method since it uses label information to form a group, whereas the standard sparse autoencoder is fully unsupervised. That being said, it is not surprising that group sparse autoencoder learns more class-specific pattern whereas sparse autoencoder doesn't. I think the fair comparison should be to autoencoders that combines classification for their objective function.
- Although authors claim that GSA learns more group-relevant features, Figure 3 (b) is not convincing enough to support this claim. For example, the first row contains many filters that doesn't look like 1 (e.g., very last column looks like 3).
- Other than visual inspection, do you observe improvement in classification using proposed algorithm on MNIST experiments?
- The comparison to the baseline model is missing. I believe the baseline model shouldn't be the sequential CNN, but the sequential CNN + sparse autoencoder. In addition, more control experiment is required that compares between the Equation (7)-(9), with different values of \alpha and \beta.
Missing reference:
Shang et al., Discriminative Training of Structured Dictionaries via Block Orthogonal Matching Pursuit, SDM 2016 - they consider block orthgonal matching pursuit for dictionary learning whose blocks (i.e., projection matrices) are constructed based on the class labels for discirminative training.