Norm matters: efficient and accurate normalization schemes in deep networks
Over the past few years, Batch-Normalization has been commonly used in deep networks, allowing faster training and high performance for a wide variety of applications. However, the reasons behind its merits remained unanswered, with several shortcomings that hindered its use for certain tasks. In this work, we present a novel view on the purpose and function of normalization methods and weightdecay, as tools to decouple weights' norm from the underlying optimized objective. This property highlights the connection between practices such as normalization, weight decay and learning-rate adjustments. We suggest several alternatives to the widely used L 2 batch-norm, using normalization in L 1 and L ∞ spaces that can substantially improve numerical stability in low-precision implementations as well as provide computational and memory benefits. We demonstrate that such methods enable the first batch-norm alternative to work for half-precision implementations. Finally, we suggest a modification to weight-normalization, which improves its performance on large-scale tasks.
This paper introduces and motivates L1 batch normalization (and variations: L_infinity batch-norm, etc.). With a theoretical study of the impact of standard (L2) batch normalization on the optimization process, the authors show that applying batch-norm makes the optimization weight-norm-invariant and is equivalent to just rescaling the learning rate (under the assumption that batch-norm is applied after a linear layer). Experiments confirm this. It is then proposed to use other kinds of normalization, namely L1, L_infinity, and a parameterized family between both, for computational efficiency reasons (faster to compute and requiring less precision, making it available for half-precision hardware), and it is shown that similar performance (in terms of accuracy) is reached as with the usual batch-norm (with experiments on ImageNet with ResNets). Finally, a similar study is conducted for weight normalization [Salimans & Kingma]. The difference with batch-norm is that the normalization is not the one of each neuronal activity over the mini-batch, but over of all input weights of each neuron together (independently of the mini-batch). As for batch-norm, It is proposed to decouple the norm of the weights from their direction, i.e. to use a representation of the kind x = rho u where u is unit-normed, for some Lp norm.  Pros: - the paper is very well written. Not just easy to read, but very well motivated, with explanations, physical meaning and intuition, etc. The style of paper I'd like to see more often. - The topic of understanding and improving batch-norm is of prime interest. Significant insights are brought. - Experiments are meaningful.   Remarks: - The analysis is specific to the case where batch-norm is applied after a linear layer. It seems this analysis should be still valid after ReLu, maxpoolings, etc. But what about after other activation functions? The experiments are performed with VGG and ResNets, which are conv-ReLu blocks. What about standard networks with really non-linear "non-linearities", i.e. not ReLu but maybe Elu, SeLu, etc.? Is the approach performance similar in practice for such non-"linear" networks? - Similarly, what about other gradients than the standard L2 one, for instance natural gradient, Adam, etc? as Adam is the most common optimizer now. Does the theoretical study still hold for such gradients? Any experiments with Adam? Actually, the paper does not state which optimizer is used, though I guess it might be SGD. - About notations: the paper is clear, but maybe some potential ambiguity in notations could be removed. For instance, sometimes the L2 norm is performed as a sum over all examples in the mini-batch; sometimes it is a sum of all input units for the considered neuron. Would it be possible to state explicitly the space over which the sum is performed? e.g. L2(mini-batch), L2(inputs). For instance line 160, or in equation 7, it is not straightforward to understand that the L2 norm stands for an expectation E_batch (and not on dimensions of x^(1) if any, e.g.). By the way, the expectation symbol would look better using \mathop{\mathbb{E}} for instance.  Edit after authors' response: I would be interested in seeing the extension of the claim to momentum methods / Adam, as it is not straightforward and would be of high interest in the community. I keep the same score.