Regularizing by the Variance of the Activations' Sample-Variances
Normalization techniques play an important role in supporting efficient and often more effective training of deep neural networks. While conventional methods explicitly normalize the activations, we suggest to add a loss term instead. This new loss term encourages the variance of the activations to be stable and not vary from one random mini-batch to the next. As we prove, this encourages the activations to be distributed around a few distinct modes. We also show that if the inputs are from a mixture of two Gaussians, the new loss would either join the two together, or separate between them optimally in the LDA sense, depending on the prior probabilities. Finally, we are able to link the new regularization term to the batchnorm method, which provides it with a regularization perspective. Our experiments demonstrate an improvement in accuracy over the batchnorm technique for both CNNs and fully connected networks.
# Post rebuttal  I believe it is a good submission (7), with some of the issues raised alraedy addressed by the authors rebuttal.   # Original review  This paper describes a new local regulariser for neural networks training to, in principle, reduce the variance of the variance of the activations.   Pros: - an easy to follow line of thought - reasonable mathematical justification of the proposed method - empirical evaluation showing results comparable with batch norm, but without all the problems BN introduces (tracking statistics, heavily affected by batch size, unclear transfer to test time etc.) - A new view of BN as a Kurtosis minimiser (while it could be obvious for some mathematicians, it is definitely a nicely described relation) - Proposed method introduces interesting multi-modality effect to neuron activations  Cons: - Evaluation is minimalistic, while batch norm is indeed extremely popular, it is not the state-of-the-art normaliser, methods like layer normalisation etc. has been shown to outperform it too, consequently it is not clear if proposed method would compare favourably or not. In reviewer's opinion, it is worth considering this paper even if the answer would be negative, as it is an alternative approach, focus on loss, rather than forward pass modification, and as such is much more generic, and "clean" in a sense.  - Change from (6) to (11) is not well justified in the text  Minor comments: - Please use \left and \right operators in equations so that brackets are of correct size (for example in (12) or (13)) - Figures require: a) bigger labels / ticks b) more precise captions, in Figure 1 what do columns represent? - please change "<
		
	
		
Reviewer 3
This paper introduces Variance Constancy Loss (VCL), a novel normalization technique for DNNs. VCL incorporates regularizer loss which penalizes variance of activation’s variance of a DNN layer. VCL allows achieving results which are better or comparable to those of Batch Normalization. These results look very interesting given the fact that VCL adds only a regularizing term and does not modify model workflow as BN does. Another benefit of VCL is that compared to BN it does not require large batches for consistency.  The experimental evaluation covers variety of UCI tasks for fully-connected networks and CIFAR-10/CIFAR-100 data for deep CNN architecture. The VCL is shown to perform better or comparable to BN in terms of final test error. However, it is not clear whether VCL speeds up the training which is a desirable feature of a normalization technique.  The paper is clearly written and easily readable. Although, it seems that there is a typo on line 146 in the inequality “n < 2N”. Instead, there should be “N < 2n” as in the following text it is said that VCL extracts two consecutive subsets of size n from a mini-batch of size N.  Overall, the paper is good. It proposes a novel normalization technique with useful properties which are theoretically grounded and are shown to be effective in practice.   Edits after Author Response:  I keep my rating. The authors have addressed my question in the rebuttal.