Removing the Feature Correlation Effect of Multiplicative Noise
Multiplicative noise, including dropout, is widely used to regularize deep neural networks (DNNs), and is shown to be effective in a wide range of architectures and tasks. From an information perspective, we consider injecting multiplicative noise into a DNN as training the network to solve the task with noisy information pathways, which leads to the observation that multiplicative noise tends to increase the correlation between features, so as to increase the signal-to-noise ratio of information pathways. However, high feature correlation is undesirable, as it increases redundancy in representations. In this work, we propose non-correlating multiplicative noise (NCMN), which exploits batch normalization to remove the correlation effect in a simple yet effective way. We show that NCMN significantly improves the performance of standard multiplicative noise on image classification tasks, providing a better alternative to dropout for batch-normalized networks. Additionally, we present a unified view of NCMN and shake-shake regularization, which explains the performance gain of the latter.
Summary: This paper presents an analysis of multiplicative noise based regularizers (e.g. DropOut) and demonstrates that employing multiplicative noise leads to internal features becoming correlated, as the regularized objective favors increasing the signal-to-noise ratio by adding redundancy. The authors propose to decompose the pre-activations of the noised layers into their signal and noise components (which is tractable since the noise under consideration is artificial) and remove the effects of the noise from the backward pass. They show that this removal changes the regularized objective to not encourage feature correlation (although it does not, as far as I can tell, penalize it, either). A connection is drawn between the proposed regularizer and shake-shake (another backward-pass-hacking multiplicative-noise based regularizer), providing (for the first time I’m aware of) some analytical justification for the latter. The results are justified through standard benchmarks on CIFAR, where they demonstrate competitive (potentially even SOTA) results for their method.  The paper is well-motivated, well-written, clear, and easy to follow. The technique proposed is simple and should be easy to implement and reproduce, without requiring more than small architectural changes. I found the analytical results convincing, and the connection to shake-shake (which, along with its variants, has thus far been an ad hoc hack) worthwhile. I found the empirical results as acceptable (the scores on CIFAR are certainly good) but lacking error bars on test error (which I don’t believe we should be letting authors get away with) or results on larger-scale tasks, particularly ImageNet (see my comments below). I think this is worth the attention of the NIPS community and vote in favor of acceptance.  General notes:  -The insight that features can be decomposed into the noise and signal (because the noise is artificial) is intriguing and clever. I found equation 15 particularly elegant.  -Please re-run your CIFAR experiments and report error bars. While common practice may be to *not* do this, I don’t think reviewers should be letting authors get away with this in situations where it should obviously be done. Even on a modest GPU these experiments can be run in under a day, so it should be no challenge to at least take the average of 3 and report variances for at least the top-scoring run.  -My primary complaint with this paper and with the shake-shake line of research is that no one, to my knowledge, has demonstrated that these techniques work at a scale beyond CIFAR. CIFAR-10 in particular has 4 times as many images per class as ImageNet, and it's known to be a regularization game, but for other tasks which are perhaps more practical, it’s not clear that these techniques provide any benefit. I don’t *fault* the authors for not running ImageNet experiments as it would be unreasonable to expect every group to test on large scale tasks, but I suspect that if the authors want to convince the broader community of the value of their work, they could easily do so by being the first to demonstrate that these techniques help on ImageNet (or that they don’t—such a negative result would also improve this paper).    -Calling it Feature-Decorrelating multiplicative noise may not be quite correct—as far as I can tell (and the authors are welcome to correct me on this front), nothing in the regularized objective actually encourages decorrelation between features. A more accurate name might be something along the lines of "Non-Correlating Multiplicative Noise."  While it appears that in practice variants 0 and 1 have lower feature correlations than the un-noised baseline, variant 2 often has slightly higher feature correlations (though lower than the MN baseline). This is not discussed anywhere in the paper that I can find, and I think it should be. I’m curious as to the authors’ opinion on this anyhow—is it because the noise is only applied once every two layers, or what?   -What exactly are the error bars in the bar graphs of e.g. figure 1? Are they representing the variance of feature correlation across runs with multiple different settings, or across different layers at a given resolution in a single run? Please state this explicitly in the paper.  -I found the description of hyperparameters confusing. The statements “the learning rate is set to either 0.03 or 0.04” implies that either different settings are used for different experiments, or grid search was used and the results were then averaged, but this is not clarified anywhere. If the authors wish to move hyperparameter details into supplementary material to save space in the body of the paper, that’s fine, but the way it is currently written is vague and might hinder reproduction.   -The authors might consider leveraging empirical successes of MN strategies as an auxiliary justification for using batchnorm—currently batchnorm is often used because “it just works,” but if one finds that FDMN is useful but only works when batchnorm is in the mix, then one might justify batchnorm as “I need FDMN and it only works with batchnorm, a combination which is justified by the analysis in the FDMN paper.”  -I briefly tried coding up FDMN-0 several different ways for WRN-40-4 and WRN-28-10 but didn’t really see a change in performance on CIFAR-100+. I didn't try very hard, so there could of course be mistakes in my implementation, and I have not factored this into my score . One implementation detail that was not clear to me was how to apply this to pre-activation style ResNets—should the noise be added at the batchnorm’d outputs before nonlinearities, without any other changes, or should the batchnorm orders be changed (an additional BN is mentioned for variants 1 and 2)?  -My implementation also made me wonder if the authors truly mean that they set the standard deviation of their uniform noise to 0.25 for CIFAR-100—this results in uniform noise U(-sqrt(3)/2, sqrt(3)/2),  which seems like a very large noise component.   -Minor:   -line 19, “However, such over-parameterized models are more prone to overfitting.” It’s not presently clear that this is actually true; the existence of a generalization gap does not necessarily imply overfitting, and models often continue to improve their validation score even as they squeeze the training loss ever closer to zero.  -line 132: I found the use of parentheses in “mean (variance)” momentarily confusing. I take it to mean “mean and variance.”  -The metric chosen for feature correlation seems reasonable, though in an ideal world it would be good to try methods particularly targeted at analyzing the similarity of internal representations in deep nets, e.g. SVCCA [NIPS 17]. The authors needn’t feel obliged to do this, though.    Response to author feedback: I feel that the authors' response has adequately addressed my concerns and my inclination is to score this paper as more of an 8 than a 7. I think that the line of work on "gradient-hacking" regularizers like Shake-Shake and ShakeDrop is interesting and potentially very valuable (the techniques show a lot of promise). Recent papers in this line of research have been rejected from the mainstream conferences, perhaps because the papers themselves were somewhat lacking. This method introduces fewer design choices (the choice of distribution and the variance of the noise) than the Shake* regularizers, which have myriad moving parts and necessitate architectural changes. As a practitioner I would much rather drop this noise into my code. To me this paper does a good job of justifying this style of technique by properly connecting it to better-understood existing methods, provides mathematical detail but doesn't fall prey to "mathiness", has good, although not quite "perfect," experimental results, and (apparently) has open source code. I argue more strongly in favor of acceptance.