LINEARLY CONSTRAINED WEIGHTS: RESOLVING THE VANISHING GRADIENT PROBLEM BY REDUCING AN- GLE BIAS
In this paper, we first identify angle bias, a simple but remarkable phenomenon that causes the vanishing gradient problem in a multilayer perceptron (MLP) with sigmoid activation functions. We then propose linearly constrained weights (LCW) to reduce the angle bias in a neural network, so as to train the network under the constraints that the sum of the elements of each weight vector is zero. A reparameterization technique is presented to efficiently train a model with LCW by embedding the constraints on weight vectors into the structure of the network. Interestingly, batch normalization (Ioffe & Szegedy, 2015) can be viewed as a mechanism to correct angle bias. Preliminary experiments show that LCW helps train a 100-layered MLP more efficiently than does batch normalization.
This paper studies the impact of angle bias on learning deep neural networks, where angle bias is defined to be the expected value of the inner product of a random vectors (e.g., an activation vector) and a given vector (e.g., a weight vector).  The angle bias is non-zero as long as the random vector is non-zero in expectation and the given vector is non-zero.  This suggests that the some of the units in a deep neural network have large values (either positive or negative) regardless of the input, which in turn suggests vanishing gradient.  The proposed solution to angle bias is to place a linear constraint such that the sum of the weight becomes zero.  Although this does not rule out angle bias in general, it does so for the very special case where the expected value of the random vector is a vector consisting of a common value.  Nevertheless, numerical experiments suggest that the proposed approach can effectively reduce angle bias and improves the accuracy for training data in the CIFAR-10 task.  Test accuracy is not improved, however.
Overall, this paper introduces an interesting phenomenon that is worth studying to gain insights into how to train deep neural networks, but the results are rather preliminary both on theory and experiments.
On the theoretical side, the linearly constrained weights are only shown to work for a very special case.  There can be many other approaches to mitigate the impact of angle bias.  For example, how about scaling each variable in a way that the mean becomes zero, instead of scaling it into [-1,+1] as is done in the experiments?  When the mean of input is zero, there is no angle bias in the first layer.  Also, what about if we include the bias term so that b + w a is the preactivation value?
On the experimental side, it has been shown that linearly constrained weights can mitigate the impact of angle bias on vanishing gradient and can reduce the training error, but the test error is unfortunately increased for the particular task with the particular dataset in the experiments.  It would be desirable to identify specific tasks and datasets for which the proposed approach outperforms baselines.  It is intuitively expected that the proposed approach has some merit in some domains, but it is unclear exactly when and where it is.
Minor comments:
In Section 2.2, is Layer 1 the input layer or the next?