FLIPOUT: EFFICIENT PSEUDO-INDEPENDENT WEIGHT PERTURBATIONS ON MINI-BATCHES
Stochastic neural net weights are used in a variety of contexts, including regularization, Bayesian neural nets, exploration in reinforcement learning, and evolution strategies. Unfortunately, due to the large number of weights, all the examples in a mini-batch typically share the same weight perturbation, thereby limiting the variance reduction effect of large mini-batches. We introduce flipout, an efficient method for decorrelating the gradients within a mini-batch by implicitly sampling pseudo-independent weight perturbations for each example. Empirically, flipout achieves the ideal linear variance reduction for fully connected networks, convolutional networks, and RNNs. We find significant speedups in training neural networks with multiplicative Gaussian perturbations. We show that flipout is effective at regularizing LSTMs, and outperforms previous methods. Flipout also enables us to vectorize evolution strategies: in our experiments, a single GPU with flipout can handle the same throughput as at least 40 CPU cores using existing methods, equivalent to a factor-of-4 cost reduction on Amazon Web Services.
In this article, the authors offer a way to decrease the variance of the gradient estimation in the training of neural networks.
They start in the Introduction and Section 2 by explaining the multiple uses of random connection weights in deep learning and how the computational cost often restricts their use to a single randomly sampled set of weights per minibatch, which results to higher-variance gradient estimatos than could be achieved otherwise. In Section 3 the authors offer to get the benefits of multiple weights without most of the cost, when the distribution of the weights is symmetric and fully factorized, by multiplying sampled-once random perturbations of the weights by a rank-1 random sign matrix. This efficient mechanism is only twice as costly as a single random perturbation, and the authors show how to efficiently parallelize it on GPUs, thereby also allowing GPU-ization of evolution strategies (something so far difficult toachieve). Of note, they provide a theoretical analysis in Section 3.2, proving the actual variance reduction of their efficient pseudo-sampling scheme. In Section 4 they provide quite varied empirical analysis: they confirm their theoretical results on four architectures; they show its use it to regularise on language models; they apply it on large minibatch settings where high variance is a main problem; and on evolution strategies.
While it is a rather simple idea which could be summarised much earlier in the  single equation (3), I really like the thoroughness and the clarity of the exposure of the idea. Too many papers in our community skimp on details and on formalism, and it is a delight to see things exposed so clearly -- even accompanied with a proof.
However, the painful part: while I am convinced by the idea and love its detailed exposure, and the gradient variance reduction is made very clear, the experimental impact in terms of accuracy (or perplexity) is, sadly,  not very convincing. Nowhere in the text did I find a clear rationale of why it is beneficial to reduce the variance of the gradient. The numerical results in Table 1 and Table 2 also do not show a clear improvement: Flipout does not provide the best accuracy. The gain in wall clock could be a factor, but would need to be measured on the figures more clearly. And the validation errors in Figure 2 for Evolution strategies seem to be worse than backprop.The main text itself also only claims performance “comparable to the other methods”.  The only visible gain is on the lower part Figure 2.a on a ConvNet.
This makes me wonder if the authors could do a better job of putting forward the actual advantages of their methods on the end-results: could wall clock measure be put more forward, to justify the extra work? This would, in my mind, strongly improve the case for publication of this article.
A few improvement suggestions:
* Could put earlier more emphasis of superiority to Local Reparameterization Trick in terms of architecture, not wait until Section 2.2 and section 4.1
*Should also put more emphasis on limitations, not wait until 3.1.
* Proposition 1 is quite straightforward, not sure it deserves a proposition, but it’s elegant to put it forward.
* Footnote 1 on re-using the matrices is indeed practical, but also somewhat surprising in terms of bias risks. Could it be explained in more depth, maybe by the random permutations of the minibatches making the bias non systematic and cancelling out?
* Theorem 1: For readability could merge the expectations on the joint distribution as E_{x, \hat \delta W} , rather than separate expectations with the conditional distributions.
* Theorem 1: could the authors provide a clearer intuitive explanation of the \beta term alone, not only as part of \alpha + \beta, especially as it plays such a key role, being the only one that does not disappear? And how do they explain their empirical observation that \beta is close to 0? Any intuition on that?
* Experiments: I salute the authors for providing all the details in exhaustive manner in the Appendix. Very commendable.
* Experiments: I like the empirical verification of the theory. Very neat to see.
Minor typo:
* page 2 last paragraph, “evolution strategies” is plural but the verbs are used in singular (“is black box”, “It doesn’t”, “generates”)