Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
We present weight normalization: a reparameterization of the weight vectors in a neural network that decouples the length of those weight vectors from their direction. By reparameterizing the weights in this way we improve the conditioning of the optimization problem and we speed up convergence of stochastic gradient descent. Our reparameterization is inspired by batch normalization but does not introduce any dependencies between the examples in a minibatch. This means that our method can also be applied successfully to recurrent models such as LSTMs and to noise-sensitive applications such as deep reinforcement learning or generative models, for which batch normalization is less well suited. Although our method is much simpler, it still provides much of the speed-up of full batch normalization. In addition, the computational overhead of our method is lower, permitting more optimization steps to be taken in the same amount of time. We demonstrate the usefulness of our method on applications in supervised image recognition, generative modelling, and deep reinforcement learning.
Inspired by batch-normalisation, the paper proposes to reparametrise neural networks by decoupling length and direction of the connection wight vectors. A theoretical analysis shows that the resulting gradients rescale the standard wight gradient and project it onto the compliment of the wight vector. It is agued that this leads to an improvement of the conditioning of the gradient.  Experiments show that the proposed reparametrisation leads to competitive results on various bench mark sets and improves the performance of different NN models.
The suggested reparametrisation and its theoretical analysis are very interesting and I enjoyed reading the paper.   However, some points in the theoretical analysis could be improved:   The paper argues that the new parametrisation improves the conditioning matrix of the gradient, but neither a strong theoretical argument nor a empirical demonstration for this are given. In line 127 it is said "Empirically, we find that w is often (close to) a dominant eigenvector of the covariance matrix C", but the correspond experiments are neither shown in the paper nor in the supplemental material.   In line 122/123 the authors claim "It has been observed that neural networks with batch normalization also have this property (to be relatively insensitive to different learning rates), which can be explained by this analysis.". However, it did not became clear to me, how the analysis of the previous sections can be directly transferred to batch normalisation.   Regarding the experimental part:  Except from the experiments on MNIST in section 5.2 each experiment is only run once (i.e. with one random initialisation). This makes an proper experimental evaluation of the different models challenging. Did the authors made sure that the observed differences were not only observes for this specific initialisation and was the same initialisation shared by all models?  Several reported runs would moreover allow for statements about statistical significant performance differences.   Why were all different methods (including the different versions of bath-normalisation) only compared in the first experiment, while the further compared only weight-normalisation against the standard parametrisation? It would be interesting to see how the other methods performed on this tasks as well.   Often for the standard parametrisation a smaller learning rate is chosen than for weigh-normalisation? Couldn't this result in the faster learning? And did larger learning rates lead to to large variance for the standard parametrisation?  Furthermore, the effective learning rates are probably different as well. It could be interesting to scale all gradients to the same length to analyse the pure effect of the changed direction of the gradients.   Where the results reported in Figure 6 gained after 200 epochs? Would have further training improved the results?   Missing references: When introducing mean-only batch normalisation the following paper could have been cited: Y. LeCun, L. Bottou, G. Orr, and K. R. Mu ̈ller. Efficient backprop. In Neural Networks: Tricks of the Trade, Lecture Notes in Computer Science, page 546. Springer Berlin/Heidelberg, 1998.  In line 268/269 it says "it is not clear how batch normalization could be applied to this model (LSTM)". Note, that recent research by Cooijmans et all (2016) on "Recurrent Batch Normalisation" tries to answer this question (https://arxiv.org/pdf/1603.09025)   minor comments:  - in equation 9:  w’ is not defined, in my opinion it took me a while to figure out that you mean the transposed vector (not the derivative), so that ww' is an outer product. -line 127: "gradient in v" -> gradient with respect to v -line 266: "enables" -> enable  - Figure2 and 6 are Tables