Stochastic Expectation Propagation
Expectation propagation (EP) is a deterministic approximation algorithm that is often used to perform approximate Bayesian parameter learning. EP approximates the full intractable posterior distribution through a set of local approximations that are iteratively refined for each datapoint. EP can offer analytic and computational advantages over other approximations, such as Variational Inference (VI), and is the method of choice for a number of models. The local nature of EP appears to make it an ideal candidate for performing Bayesian learning on large models in large-scale dataset settings. However, EP has a crucial limitation in this context: the number of approximating factors needs to increase with the number of datapoints, N , which often entails a prohibitively large memory overhead. This paper presents an extension to EP, called stochastic expectation propagation (SEP), that maintains a global posterior approximation (like VI) but updates it in a local way (like EP). Experiments on a number of canonical learning problems using synthetic and real-world datasets indicate that SEP performs almost as well as full EP, but reduces the memory consumption by a factor of N . SEP is therefore ideally suited to performing approximate Bayesian learning in the large model, large dataset setting.
QUALITY This is an interesting idea. However, I think the paper requires two major changes to be interesting to a NIPS reader.First, the paper should be up front about its experimental nature and follow with an in-depth study. Specifically, the experiments should clearly highlight the strengths (big datasets) and weaknesses (clear, thorough comparisons and failure cases).Second, the paper requires some editorial work. The important sections (see below) could use more explanation. The experimental study should clearly motivate the setup and carefully discuss the results with technical detail. This is what a NIPS reader would want to get from such a paper.Some details:Bayesian Probit Regression: I do not understand why this approximate KL is a good metric to report. Why not just compare means and variances of parameter estimates as compared to Stan/NUTS? Isn't that much more straightforward and natural?Figure 3(c): I am confused as to why SEP with K=1 is compared to DSEP with K=10. Am I missing something here?Mixture of Gaussians: why did you now switch to this F-norm metric? Also wouldn't it be better to simulate some data where EP fails to recover the truth? I would rather want to see whether SEP "fails in the same way" as EP, or whether the single f(\theta) factor provides a different behavior.Having reached Section 5.3, I am a bit dissatisfied by what these first two experimental sections present. I have not learned much about the nature of SEP: specifically, I don't understand how it fails. The investigation of minibatch sizes is limited, as the experiments do not paint a clear picture. Also, given that the main motivation of this work is to scale EP, I don't understand why the majority of the experiments focus on tiny datasets.Section 5.3 requires significant overhaul. The experimental setup is brushed aside with a citation: this is unacceptable. The results are discussed in a completely conversational manner, with little technical detail or specificity, using words like "often", "interestingly", "possibly", and "likely".CLARITY Figure 3(b): DAEP is never discussed, either in the caption or in the text.I am not sure what Figure 2 adds to the paper. I would rather the authors expand on Sections 3 and 4.1: both of these sections are important for the paper, and yet they would benefit from a clearer exposition. Removing Figure 2 would afford this space to clarify these important ideas of the paper.ORIGINALITY The idea is nice. It is also original: i have not seen any work that addresses EP's memory constraints. The proposal is straightforward; thus a NIPS reader will immediately ask: "that's a good idea, I wonder why it works."SPECIFICS line 48: ... having A myriad OF local ...line 69: truelyline 149: summariedline 199: arguableline 214: this paragraph is vague. I cannot tell what you are referring to without looking at the supplementary material; thus I have little motivation to look it up in the supplementary material.line 257: why is "mixture" capitalized?figure 2 should appear on page 5, not page 6.line 290: i am thoroughly confused by this comparison metric.line 292: there is no citation numbered [25]. should it be [24]? This makes me worried about all of the citations in the rest of the paper...