A widely observed phenomenon in deep learning is the degradation problem: increasing the depth of a network leads to a decrease in performance on both test and training data. Novel architectures such as ResNets and Highway networks have addressed this issue by introducing various flavors of skip-connections or gating mechanisms. However, the degradation problem persists in the context of plain feed-forward networks. In this work we propose a simple method to address this issue. The proposed method poses the learning of weights in deep networks as a constrained optimization problem where the presence of skip-connections is penalized by Lagrange multipliers. This allows for skip-connections to be introduced during the early stages of training and subsequently phased out in a principled manner. We demonstrate the benefits of such an approach with experiments on MNIST, fashion-MNIST, CIFAR-10 and CIFAR-100 where the proposed method is shown to greatly decrease the degradation effect and is often competitive with ResNets.
EDIT: The rating has been changed. See thread below for explanation / further comments.
ORIGINAL REVIEW: In this paper, the authors present a new training strategy, VAN, for training very deep feed-forward networks without skip connections (henceforth called VDFFNWSC) by introducing skip connections early in training and then gradually removing them. 
I think the fact that the authors demonstrate the viability of training VDFFNWSCs that could have, in principle, arbitrary nonlinearities and normalization layers, is somewhat valuable and as such I would generally be inclined towards acceptance, even though the potential impact of this paper is limited because the training strategy proposed is (by deep learning standards) relatively complicated, requires tuning two additional hyperparameters in the initial value of \lambda as well as the step size for updating \lambda, and seems to have no significant advantage over just using skip connections throughout training. So my rating based on the message of the paper would be 6/10.
However, there appear to be a range of issues. As long as those issues remain unresolved, my rating is at is but if those issues were resolved it could go up to a 6.
+++ Section 3.1 problems +++
- I think the toy example presented in section 3.1 is more confusing than it is helpful because the skip connection you introduce in the toy example is different from the skip connection you introduce in VANs. In the toy example, you add (1 - \alpha)wx whereas in the VANs you add (1 - \alpha)x. Therefore, the type of vanishing gradient that is observed when tanh saturates, which you combat in the toy model, is not actually combated at all in the VAN model. While it is true that skip connections combat vanishing gradients in certain situations, your example does not capture how this is achieved in VANs.
- The toy example seems to be an example where Lagrangian relaxation fails, not where it succeeds. Looking at figure 1, it appears that you start out with some alpha < 1 but then immediately alpha converges to 1, i.e. the skip connection is eliminated early in training, because wx is further away from y than tanh(wx). Most of the training takes place without the skip connection. In fact, after 10^4 iterations, training with and without skip connection seem to achieve the same error. It appears that introducing the skip connection was next to useless and the model failed to recognize the usefulness of the skip connection early in training.
- Regarding the optimization algorithm involving \alpha^* at the end of section 3: It looks to me like a hacky, unprincipled method with no guarantees that just happened to work in the particular example you studied. You motivate the choice of \alpha^* by wanting to maximize the reduction in the local linear approximation to \mathcal{C} induced by the update on w. However, this reduction grows to infinity the larger the update is. Does that mean that larger updates are always better? Clearly not. If we wanted to reduce the size of the objective according to the local linear approximation, why wouldn't we choose infinitely large step sizes? Hence, the motivation for the algorithm you present is invalid. Here is an example where this algorithm fails: consider the point (x,y,w,\alpha,\lambda) = (100, \sigma(100), 1.0001, 1, 1). Here, w has almost converged to its optimum w* = 1. Correspondingly, the derivative of C is a small negative value. However, \alpha* is actually 0, and this choice would catapult w far away from w*.
If I haven't made a mistake in my criticisms above, I strongly suggest removing section 3.1 entirely or replacing it with a completely new example that does not suffer from the above issues.
+++ ResNet scaling +++
There is a crucial difference between VANs and ResNets. In the VAN initial state (alpha = 0.5), both the residual path and the skip path are multiplied by 0.5 whereas for ResNet, neither is multiplied by 0.5. Because of this, the experimental results between the two architectures are incomparable.
In a question I posed earlier, you claimed that this scaling makes no difference when batch normalization is used. I disagree. Let's look at an example. Consider ResNet first. It can be written as x + r_1 + r_2 + .. + r_B, where r_b is the value computed by residual block b. Now let's assume we insert a scaling constant after each residual block, say c = 0.5. Then the result is c^{B}x + c^{B-1}r_1 + c^{B-2}r_2 + .. + r_B. Therefore, contributions of lower blocks vanish exponentially. This effect is not combated by batch normalization.
So the learning dynamics for VAN and ResNet are very different because of this scaling. Therefore, there is an open question: are the differences in results between VAN and ResNet in your experiments caused by the removal of skip connections during training or by this scaling? Without this information, the experiments have limited value. In fact, I suspect that the vanishing of the contribution of lower blocks bears more responsibility for the declining performance of VAN at higher depths than the removal of skip connections.
If my assessment of the situation is correct, I would like to ask you to repeat your experiments with the following two settings: 
- ResNet where after each block you multiply the result of the addition by 0.5, i.e. x_{l+1} = 0.5\mathcal{F}(x_l) + 0.5x_l
- VAN with the following altered equation: x_{l+1} = \mathcal{F}(x_l) + (1-\alpha)x_l, i.e. please remove the alpha in front of \mathcal{F}. Also, initialize \alpha to zero. This ensures that VAN starts out as a regular ResNet.
+++ writing issues +++
Title:
- "VARIABLE ACTIVATION NETWORKS: A SIMPLE METHOD TO TRAIN DEEP FEED-FORWARD NETWORKS WITHOUT SKIP-CONNECTIONS" This title can be read in two different ways. (A) [Train] [deep feed-forward networks] [without skip-connections] and (B) [Train] [deep feed-forward networks without skip connections]. In (A), the `without skip-connections' modifies the `train' and suggests that training took place without skip connections. In (B), the `without skip-connections' modifies `deep feed-forward networks' and suggests that the network trained has no skip connections. You must mean (B), because (A) is false. Since it is not clear from reading the title whether (A) or (B) is true, please reword it.
Abstract:
- "Part of the success of ResNets has been attributed to improvements in the conditioning of the optimization problem (e.g., avoiding vanishing and shattered gradients). In this work we propose a simple method to extend these benefits to the context of deep networks without skip-connections." Again, this is ambiguous. To me, this sentence implies that you extend the benefit of avoiding vanishing and exploding gradients to fully-connected networks without skip connections. However, nowhere in your paper do you show that trained VANs have less exploding / vanishing gradients than fully-connected networks trained the old-fashioned way. Again, please reword or include evidence.
- "where the proposed method is shown to outperform many architectures without skip-connections" Again, this sentence makes no sense to me. It seems to imply that VAN has skip connections. But in the abstract you defined VAN as an architecture without skip connections. Please make this more clear.
Introduction:
- "Indeed, Zagoruyko & Komodakis (2016) demonstrate that it is better to increase the width of ResNets than the depth, suggesting that perhaps only a few layers are learning useful representations." Just because increasing width may be better than increasing depth does not mean that deep layers don't learn useful representations. In fact, the claim that deep layers don't learn useful representations is directly contradicted by the paper.
section 3.1:
- replace "to to" by "to" in the second line
section 4:
- "This may be a result of the ensemble nature of ResNets (Veit et al., 2016), which does not play a significant role until the depth of the network increases." The ensemble nature of ResNet is a drawback, not an advantage, because it causes a lack of high-order co-adaptataion of layers. Therefore, it cannot contribute positively to the performance or ResNet.
As mentioned in earlier comments, please reword / clarify your use of "activation function". It is generally used a synonym for "nonlinearity", so please use it in this way. Change your claim that VAN is equivalent to PReLU. Please include your description of how your method can be extended to networks which do allow for skip connections.
+++ Hyperparameters +++
Since the initial values of \lambda and \eta' are new hyperparameters, include the values you chose for them, explain how you arrived at those values and plot the curve of how \lambda evolves for at least some of the experiments.