content_image/314725.jpg content_image/314726.jpg
fore approximate the ELBO objective function L using a new objective function Q L . To construct Q L , we consider two alternatives: the zeroth-order and first-order delta method for moments (Bickel and Doksum 2007), which we call D0 and D1 respectively. D0 is equiva- lent to applying Jensen’s inequality to the expected log-sum-exp, resulting in the following lower bound to (22): 

<img src='content_image/314725.jpg'>

Here we used the usual formula for the mean of a lognormal random variable. In a different context, Blei and Lafferty (2007) consider an approximation equivalent to D0 but expressed using a redundant variational parameter. 

For approximation D1, we restrict 6$_{h}$ GLYPH<31> 0 to be diagonal, and define GLYPH<27>$_{h}$ : D log . diag f 6$_{h}$ g / 2 R $^{K}$. Using results in Appendix B, we obtain the following approx- imation to (22): 

<img src='content_image/314726.jpg'>

with 2.GLYPH<22>$_{h}$ / 2 R K as defined in Appendix B. Notice that, unlike D0, approximation D1 does not preserve the guarantee that the optimal value of the variational optimization lower bounds the marginal likelihood. However, in our simulations, using D1 resulted in more accurate variational approximations to the posterior. 

In this appendix we give a derivation based on approximation D0. The derivation for D1 is similar, but simpler, because 6$_{h}$ is treated as diagonal. Under D0, the final empirical 

