Reducing Reparameterization Gradient Variance
Optimization with noisy gradients has become ubiquitous in statistics and machine learning. Reparameterization gradients, or gradient estimates computed via the "reparameterization trick," represent a class of noisy gradients often used in Monte Carlo variational inference (MCVI). However, when these gradient estimators are too noisy, the optimization procedure can be slow or fail to converge. One way to reduce noise is to generate more samples for the gradient estimate, but this can be computationally expensive. Instead, we view the noisy gradient as a random variable, and form an inexpensive approximation of the generating procedure for the gradient sample. This approximation has high correlation with the noisy gradient by construction, making it a useful control variate for variance reduction. We demonstrate our approach on a non-conjugate hierarchical model and a Bayesian neural net where our method attained orders of magnitude (20-2,000×) reduction in gradient variance resulting in faster and more stable optimization.
Summary
This paper proposes a control variate (CV) for the reparametrization gradient by exploiting a linearization of the data model score.  For Gaussian random variables, such a linearization has a distribution with a known mean, allowing its use as a CV.  Experiments show using the CV results in faster (according to wall clock time) ELBO optimization for a GLM and Bayesian NN.  Furthermore, the paper reports 100 fold (+) variance decreases during optimization of the GLM.  
Evaluation
Method:  The CV proposed is clever; the observation that the linearization of the data score has a known distribution is non-obvious and interesting.  This is a contribution that can easily be incorporated when using the reparametrization trick.  The only deficiencies of the method are (1) it requires the model’s Hessian and (2) application is limited to Gaussian random variables.  It also does not extend to amortized inference (i.e. variational autoencoders), but this limitation is less important.  I think the draft could better detail why it’s limited to Gaussian variables (which is a point on which I elaborate below under ‘Presentation’).  Some discussion of the hurdles blocking extension to other distributions would improve the draft and be appreciated by readers.  For instance, it looks like the linearized form might have a known distribution for other members of the location-scale family (since they would all have the form f(m) + H(m)(s*epsilon))?  Thoughts on this?    
Experiments:  Experiments are limited to small models (by modern standards)---GLM and one-layer / 50 hidden unit Bayesian NN---but are adequate to demonstrate the correctness and utility of the method.  I would have liked to see what held-out performance gains can be had by using the method---you must have looked at held-out LL?---but I realize this is not crucial for assessing the method as a CV.  Still I think the paper is strong enough that even a negative or neutral result would not hurt the paper’s quality.
Presentation:  The paper’s presentation of the method’s derivation is clear and detailed.  I had no trouble seeing the logical flow from equation to equation.  However, I think the draft’s description is somewhat ‘shallow’---by that I mean, while the algebra is clear, more discussion and intuition behind the equations could be included.  Specifically, after the method is derived, I think a section discussing why it's limited to Gaussian rv’s and why it can’t be extended to amortized inference would be beneficial to readers.  It took me two-to-three reads through the draft before I fully understood these limitations.  To make room for this discussion, I would cut from the introduction and VI background since, if a person is interested in implementing your CV, they likely already understand VI, the ELBO, etc. 
Minutiae: Table 1 is misreferenced as ‘Table 3’ in text.
Conclusion:  This paper was a pleasure to read, and I recommend its acceptance.  The methodology is novel and interesting, and the experiments adequately support the theory.  The draft could be improved with more exposition (on limitations, for instance), but this is not a crucial flaw.