Copula variational inference
We develop a general variational inference method that preserves dependency among the latent variables. Our method uses copulas to augment the families of distributions used in mean-field and structured approximations. Copulas model the dependency that is not captured by the original variational distribution, and thus the augmented variational family guarantees better approximations to the posterior. With stochastic optimization, inference on the augmented distribution is scalable. Furthermore, our strategy is generic: it can be applied to any inference procedure that currently uses the mean-field or structured approach. Copula variational inference has many advantages: it reduces bias; it is less sensitive to local optima; it is less sensitive to hyperparameters; and it helps characterize and interpret the dependency among the latent variables.
The typical mean-field approximation made in variational inference results in a well-known range of issues, from poor approximation of the distribution as a whole to poor covariance estimates in particular. This is a pressing problem for variational inference, and the authors aim to address it by introducing copulas into the variational approximating distribution. The authors' experimental results suggest that this may be a promising direction for improving on variational approximations. However, some details of the method are not covered as much as desired, and the experiments seem somewhat limited.Major points(1) In order to run their method, the authors need to choose a vine and a copula. While the authors spend a little time discussing that they choose these according to an algorithm, it's not clear what the approximative significance of the vine in particular is. Why are some vines better than others? Why are some copulas better than others? Are all possible vines considered? Is that problematic in many-parameter models? How are the possible copulas chosen? Is there any reason to think these choices of copulas are enough? E.g., on the tree point,p. 4: line 163--168: Where does the first tree come from? Can it be any tree on the model variables? Does it matter?p. 5, line 243: What does it mean to learn the tree structure? What is there to learn?(2) The authors seem to use one model per performance metric: a Gaussian mixture model for variance modeling and a latent state space model for predictive likelihood. (Why not compare both models on both metrics?) This isn't necessarily bad; it could be a starting point for future exploration. But then the authors seem to make somewhat broad conclusions based on one model each. It's also unclear why the authors plot just one particular set of parameters from the Gaussian mixture model in Figure 3. Aren't there are other mixture model parameters (like the component means in Eq. 18)? For that matter, what are the hyperparameters in the each generative model (the ones used in the experiment)?p. 8, line 393--396: It seems from Fig 3 that CVI and LRVB both estimate variances equally well on this problem. LRVB focuses on covariances. Presumably the point of CVI is to get an entire posterior that is better. Table 1 gets at this a little, but it would be nice to see more comparisons on, say, estimates of other posterior statistics than just covariances. Or on other models. And why not compare to structured mean-field or other recent methods that get beyond mean-field for predictive likelihood?p. 8, line 418: "CVI always outperforms LRVB". This statement seems a bit broad to be warranted from the experiments presented. The authors might reasonably say CVI outperforms LRVB in predictive likelihood (based on Table 1)---but does it outperform across other models besides just the one mixture model presented? Or do the authors really mean to say it outperforms LRVB in other ways? It is unclear how the two compare in covariance estimation (besides looking roughly the same in Figure 3).Minor pointsp. 1, line 35--37: This sentence is a little difficult to parse on the first pass. It sounds like the authors are saying it's a generalization but also a special case. Presumably they are just saying that their algorithm iterates between mean-field steps and copula steps. Does "structured factorizations" here refer to [19]? If so, it would help the reader to be explicit.p. 2, line 83: It seems a little unusual of a notational choice to use z for all latent variables; it seems to be usually reserved for indicatorsp. 2, line 84: KL is technically not a distance since it is not symmetricp. 3, line 143: "the bivariate Gaussian distribution with zero mean and Pearson correlation coefficient $\rho$" <-- Does this distribution have arbitrary strictly positive variance parameters in both directions? (or all the variances one since the authors refer to "the bivariate Gaussian"?)p. 5, line 247--248: What does this mean; e.g., what are these "future iterations"? Are the authors saying that the choice of tree and copula family is stable across multiple independent runs of the algorithm? If so, what changes from run to run that this choice might be different? If not, what are the iterations? What is the significance of the lack of change described?p. 7, line 357: "reduce the variance" <-- The authors might want to use more careful language here to clarify what variance is being reduced since the word "variance" is being used in a few different ways around this point in the text.p. 8, Table 1 seems to be missing running time. It's in the text, but it's a little hard to parse the comparison in paragraph form. Is there any reason not to put it in the table?p. 8, line 389: "consistent estimates" <-- This seems to be a poor use of the word "consistent" in the statistical sense (there's no limit as the number of data points becomes infinite). Perhaps "accurate" is the desired word.p. 8, line 412: This paragraph is a little hard to read. It sounds like CVI took hours, but then the authors say it's better in runtime. Perhaps this could be clarified.------------------------------ After Author Feedback ------------------------------The author feedback seemed on point; I look forward to reading more detail about the vine methodology and a broader range of experiments in the final version.