The Population Posterior and Bayesian Modeling on Streams
Many modern data analysis problems involve inferences from streaming data. However, streaming data is not easily amenable to the standard probabilistic modeling approaches, which require conditioning on finite data. We develop population variational Bayes, a new approach for using Bayesian modeling to analyze streams of data. It approximates a new type of distribution, the population posterior, which combines the notion of a population distribution of the data with Bayesian inference in a probabilistic model. We develop the population posterior for latent Dirichlet allocation and Dirichlet process mixtures. We study our method with several large-scale data sets.
- Summary of Paper   - The paper describes the development of an evidence lower bound     (ELBO) constructed by averaging over a data-generating     distribution. The paper shows that optimizing this ELBO leads to     impressive results on very large data streaming applications. - Quality   - L039: "the standard approach to probabilitic modeling", might be     better stated as "the standard approach to Bayesian probabilitic     modeling" since we can build a probabilitic model without a data     set in hand.   - L044: On initial reading, my reaction was, "Why shouldn't the     Bayesian posterior become more confident with more data? Indeed,     this is a desirable property of Bayesian inference procedures.     Also, "over-dispersion" is a well-known problem for many     generalized linear models and the typical solution there is to     build a better model. So, isn't the solution here, to build a     better model? Or if there is uncertainty about the model, perhaps     we should average over models in some way." However, later, some     clarity is provided in that the procedure aims to be robust to     model specification in a different way.   - L051: Again, my initial reaidng of this paragraph caused me to be     concerned that the real problem with Bayesian updating on data     streams is not the Bayesian procedure, but the way the model has     been specified. If the data stream is changing and we haven't     explicitely modeled that, then of course the updates may yeild     poor inferences, but that's not because our updating procedure is     flawed, but because our model is flawed. Here again, it seems     that the proposed procedure is trying to be robust to model     specification issues that really cause problem on data streams.     Perhaps the narrative in these introductory paragraphs can be     sharpened to set up the nice work presented later.   - L056: The claim is that explicitly modeling the time series     incurs a heavy inferential cost. Can this claim be supported with     a citation or other evidence?   - L165: Is there a misplaced parenthesis and perhaps a missing     \beta in the variantional distribution in the F-ELBO?   - L165: The standard ELBO is conditional on a particular data set x     and the F-ELBO is an average over data set x provided by the     population distribution X ~ F_\alpha. I'm curious if taking this     average causes the F-ELBO to preferentially optimize the ELBO     over modes of F_\alpha. Whereas, if we conditioned on a     particular x, as in the ELBO, it wouldn't matter how likely that     data set is under F_\alpha. Can the authors comment on the     tradeoffs of marginalizing over F_\alpha versus conditioning on a     sample from it?   - The results primarily deal with prediction rather than parameter     estimation. This is entirely appropriate given the applications     where streaming data is typically found. However, is there     anything that can be said about the parameter estimates,     especially given the first-order goal of maximizing the ELBO or     F-ELBO is to obtain parameter estimates?   - I do like that the F-ELBO explains SVI and provides a nice     framework for understanding that sampling procedure. But, I     wonder if one has in hand a generative model for p(X), what is     the costs/benefits of using that distribution as an averaging     distribution instead of X ~ F_\alpha? I understand that if our     model is misspecified, averaging with respect to that model may     exacerbate the updating problems outlined, and instead drawing     samples from F_\alpha is model-independent. Is there any guidance     as to another reason p(X) is a poor choice? - Clarity   - It would help to clarify exactly where the problems identified in     paragraph 2 and 3 in the introduction lie. L042 says that the     problems are with "Bayesian updating on data streams", but L044     says "the first problem is that Bayesian posteriors will become     overconfident" and L051 says "the data stream might change over     time". After reading these assertions several times, it becomes     clear what is intended, but I think the statement on L044 could     be better as "The first problem with Bayesian updating on data     streams is that Bayesian posteriors will become overconfident"     and L051 could be "The second problem with Bayesian updating on     data streams is that the data stream might change over time     causing ..." - Originality   - The paper is original and provides a good justification for SVI. - Significance   - I find the paper to be highly significant and I hope will be a     welcome addition in the community.