Variational Dropout and the Local Reparameterization Trick
We investigate a local reparameterizaton technique for greatly reducing the variance of stochastic gradients for variational Bayesian inference (SGVB) of a posterior over model parameters, while retaining parallelizability. This local reparameterization translates uncertainty about global parameters into local noise that is independent across datapoints in the minibatch. Such parameterizations can be trivially parallelized and have variance that is inversely proportional to the minibatch size, generally leading to much faster convergence. Additionally, we explore a connection with dropout: Gaussian dropout objectives correspond to SGVB with local reparameterization, a scale-invariant prior and proportionally fixed posterior variance. Our method allows inference of more flexibly parameterized posteriors; specifically, we propose variational dropout, a generalization of Gaussian dropout where the dropout rates are learned, often leading to better models. The method is demonstrated through several experiments.
** Updated review after authors' response **I spent some time trying to get a broader view of the work since I wrote my review. To be honest, my view now is that the paper is very well written, but with almost no novel contribution. The main idea is "sample a new random variable realisation for each data point in the mini-batch". I have been using this technique for most of my applications working with SVI - it is fairly obvious (although the theoretical variance analysis is novel as far as I know). It does have some computational limitations that the authors avoid by the re-parametrisation trick - which I'm afraid is *very* problem specific. I can't easily see how this trick could be used for models other than MLPs. The author's answer "We show that dropout can be interpreted as being as a result of, rather than being equal to, a local reparameterization of weight uncertainty" suggests that the authors claim dropout to be a special case of the technique; this would have been fine if other, non-trivial, models were used as well. This is my main concern, that the paper would have been *a lot stronger* (thus beneficial to the community) if the authors were to suggest and assess this with general models (rather than models that make use of it anyway, see next). The authors do mention in the rebuttal that this should be applicable to "a larger family of graphical models with plates" which I think is what we would have been most interested in. My second major concern is that the models used in the assessment use this trick anyway. Even though the authors claim in the rebuttal that earlier methods do not use the technique, I am certain that all implementations of dropout (with all its variants) sample new random variable realisations for each point in the mini-batch. I spent a lot of time studying different open-source implementations and, even though this is sometimes overlooked in the literature, the actual implementation would do this. The dropout journal paper mentions this explicitly, and the popular Caffe open source framework implements this as well. Another example is the increasingly popular Keras framework which implements Gaussian multiplicative noise as well as other noise layers, all of which sample new realisations for each data point (https://github.com/fchollet/keras/blob/master/keras/layers/noise.py). I would note that fast dropout operates differently and matches the first two moments of the input. That's why I find the experiments not satisfying. The dropout MLP models are presumably just the vanilla models used in the wild. The main improvement is the optimisation of the dropout parameter. But this was already done before in [A Bayesian Encourages Dropout, Maeda] and can't be counted as a novel contribution (for that matter Shin-ichi also discusses a Bayesian interpretation of dropout). The novel contributions of the paper seem to be1. "sample separate random variable realisations per data-point in stochastic variational inference to reduce the estimator variance" (which I think is important, but somewhat known) and2. an explanation for dropout's low variance optimisation objective (although it might be just my interpretation as I believe this is not mentioned explicitly as an insight for dropout from the analysis).After a long deliberation I have decided to reduce my score slightly as the paper does not communicate these contributions well in its current form. Even though the paper is written well (in the sense that it is very pleasant to read), both the authors and the community would have benefited greatly from re-structuring the paper to present these more focused ideas. ** Review before authors' response **  The main results of the work are very interesting. The analysis (both theoretical and experimental) of the estimator variance of both fast dropout and multiplicative Gaussian noise dropout is important and enlightening.  Major criticism: * The main contribution of the paper is linking multiplicative Gaussian noise dropout and fast dropout to variance reduction techniques in a Bayesian setting, as is mentioned explicitly in the introduction ("in this paper we show that a relationship between dropout and Bayesian inference can be extended and exploited [...]").This is stretched rather thinly as a new technique for gradient based variational inference ("we explore an as yet unexploited trick for drastically improving the efficiency of stochastic [...]") even though the "trick" mentioned is known and has been used for quite some time: both multiplicative Gaussian noise dropout and fast dropout perform this, and this is not new by itself. I would have expected to see the authors use this technique for new models in the experiments section rather than presenting this as a new development and then assessing this with models that *already* use this technique.* I'm confused about the generality of the method suggested. In line 107 it is mentioned that the method can be applied to any probabilistic model, which is true for sampling a new random variable for each data point (as is often done anyway). But the computationally efficient form in line 171-180 is only applicable to neural networks. Furthermore, experiments are only done for neural networks, with all models compared simply being multiplicative Gaussian noise dropout and fast dropout with the only novelty being the tuning of the dropout parameter. * section 3.3: the use of a log-uniform prior seems quite forced (mostly to get rid of the KL divergence term as a function of theta). Especially given that dropout is often used with L2 regularisation in many models, which would add additional unaccounted-for theta terms to the objective function. Furthermore, the fact that the authors have to limit the value of alpha (section 3.4) shows that the effects of the regularisation are not sufficient. Minor criticism: * The citation format is incorrect. * Lines 185-186: It is not clear whether by a "local re-parameterisation trick" the authors mean sampling a separate random variable for each data point in the mini-batch, or specifically sampling a separate random variable for each *unit* in a network. * Lines 57-59: this is wrong; [1] has shown identical results to dropout in a Bayesian setting. * Line 144: why is cov positive on average?* Line 404: why wasn't the dropout probability p scaled down for the smaller networks? what are the alpha values used with fast dropout and multiplicative Gaussian noise dropout? * Lines 410-413: why wasn't Variational (A2) assessed for CIFAR-10 as well? I hope the authors will find the following minor comments helpful: * Many citations have proper names in lower case (such as fisher, bayesian, langevin, etc). You can fix this by surrounding these words with curly brackets in the bib file ("... fisher ..." -> "... {F}isher ..."). * Line 61: "data(Kingma" * Line 100: g() should be f() * Line 112: "are correct on average" is quite informal * Line 132: why does the equation contain the term L_D(phi)^2/N^2 instead of E(L_i)^2? * Line 215: "with a M x K of..." should be "with an M x K matrix of..." * Line 216: "by adding noise ... the weights are less likely to overfit" - why?* Line 233: Wang and Manning (2013)'s derivation uses Bernoullis on the inputs to the product, and the marginal was *approximately* Gaussian. Presumably the reason for the good performance for both dropout and fast dropout is that both Gaussians and Bernoullis had the low variance discussed in this paper (compared to distributions over the weights); fast dropout simply approximated the output of the Bernoulli times weights. * Line 239: "activation noise" is not defined * Line 255-257: a bit confusing * Line 262: "... for which the noise enters this way" is not clear * Lines 290-292: can be moved to appendix * Line 303: "performance is quite robust to different specifications of alpha" - presumably because the model will simply adapt the weights to match the moments of the full posterior? * Line 318: \ref{} should be \eqref{} for equation 3 * Line 330: "similar to our local reparameterization trick" - not so much; Bayer simply matches the first two moments of the input distribution in a variational setting, equivalent to fast dropout with generalised noise processes. * Section 4: you might consider adding references to [2], [3] * Line 362: "lower" should be "lowest" * Lines 371-373: it is not clear that the three models correspond to random variable samples per unit / samples per weight / samples per batch (or data point?) References:[1] Weight Uncertainty in Neural Network Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, Daan Wierstra, ICML, 2015 http://jmlr.org/proceedings/papers/v37/blundell15[2] A Bayesian Encourages Dropout Shin-ichi Maeda Under review as a conference paper at ICLR 2015 http://arxiv.org/pdf/1412.7003.pdf[3] Dropout As A Bayesian Approximation: Insights And Applications Yarin Gal, Zoubin Ghahramani Deep Learning Workshop, ICML, 2015https://sites.google.com/site/deeplearning2015/33.pdf?attredirects=0