Importance Weighting and Variational Inference
Recent work used importance sampling ideas for better variational bounds on likelihoods. We clarify the applicability of these ideas to pure probabilistic inference, by showing the resulting Importance Weighted Variational Inference (IWVI) technique is an instance of augmented variational inference, thus identifying the looseness in previous work. Experiments confirm IWVI's practicality for probabilistic inference. As a second contribution, we investigate inference with elliptical distributions, which improves accuracy in low dimensions, and convergence in high dimensions.
Update: in light of Reviewer One's observation that Theorem 3 is a special case of Proposition 1(c) of the FVO paper, and that Theorem 1 a version -- after some manipulation -- of Claim 1 in the AESMC paper, I will downgrade my review from a 7 to a 6.  -----------  The paper provides semantics for the posterior approximation that is implicit in Importance Weighted Autoencoders [4].  In particular, their key result is that by maximizing the heuristically motivated importance weighted ELBO (IW-ELBO) from [4] is equivalent to minimizing a particular KL divergence (Theorems 1 and 2).  This KL divergence is not directly between an approximation and the target posterior p(z | x), but contains an additional term that quantifies the effect of importance sampling.  However, this insight motivates the use of the optimizer of the IW-ELBO to compute improved estimates of posterior expectations (equation 9).  The paper is well-written and well-motivated, and provides valuable semantics to the IW-ELBO, and I recommend that it be accepted.  I have some concerns about the the importance weighting procedure, however -- in particular, about whether or not the weights (denoted R) can be expected to have finite variance.  To be fair, these concerns also extend to the original published work [4] to some extent, although [4] skirts the problem somewhat by only concerning itself with log(R) (appendix B of [4]), whereas this paper relies on assertions about R itself.  Specifically, the claim that R_M places less mass near zero for larger M (line 101) does not seem obvious to me in cases where equation (4) has infinite variance.  Similarly, the usefulness of Theorem 3 depends on R having finite variance.  It seems that additional conditions would need to hold to ensure that equation (4) has finite variance -- see Owen [13], example 9.1 and related discussion.  The original reference Burda et al. [4] does not seem to address this problem -- they only control the mean absolute of the E[log R_M], certainly not the variance of R_M.  Note also that Burda et al. [4] Theorem 1 guarantees only that the bound does not degrade, but the authors claim that the bound in fact improves.  Given the tendency of vanilla KL(q || p) to have q very small where p is large, I’d like to see this potential issue addressed more directly.  (At line 165 the authors say that “the variance of R is a well-explored topic in traditional importance sampling”, which is true, but I think it is still important to address the variance of R in this particular context.)  In particular, I wonder whether this method could work when you don’t allow the variational parameterization to have a full covariance matrix (that is, a non-diagonal A_w in equation 14).  Using a full covariance matrix is only feasible in relatively small dimensions because the number of free parameters in A_w scales quadratically with dimension.  Notably, most of the experiments are all relatively low dimensional -- the clutter and dirichlet models use no more than a 10-dimensional parameter space, and even the largest “large” logistic regression model has only up to a 500-dimensional parameter.  (Notably the authors do not report computation time for the logistic model.)   One metric that would be revealing would be the entropy of the normalized weight distribution.  Are a relatively few data points getting most of the weight after normalization?  If so, it’s an indication that the importance sampling procedure is not working well.  A relatively even distribution of normalized weights throughout the optimization process would be reassuring that importance weighting is doing what the authors claim.  It would be more interesting to compare the logistic regression model posterior moments to an MCMC ground truth as with the other models.  Minor typos: - Typo in the definition of R in line 20 -- there should be no log. - The bibliography looks strange.  What are the trailing numbers? -  Line 109: Can be understood - Algorithm 1.  The weight function and q should be explicit or an input.  (I think for theorem 1 to hold they need to be the importance weights of equation 3, and it should be clearer that they are not general.) - Theorem 1. Please define things before you refer to them (specifically p_M) -  Line 184 “y” should probably be “u” - Line 191: It would probably be better to avoid double use of the letter w for both weights and variational parameters - Line 220: gradient -> gradients - I find the notation “E-IWVI” vs “IWVI” (line 231) to be a bit strange.  There doesn’t seem to be anything about the importance weighting trick that requires a Gaussian approximation.  So using the unmodified “IWVI” to refer to a Gaussian q and the modified “E-IWVI” to refer to an elliptical q is confusing.  Perhaps “G-IWVI” could refer to the Gaussian q, and the unmodified “IWVI” can go back to referring to the importance weighting trick irrespective of the choice of q. - Double periods in the caption of figure 5.