A Probabilistic Programming Approach To Probabilistic Data Analysis
Probabilistic techniques are central to data analysis, but different approaches can be challenging to apply, combine, and compare. This paper introduces composable generative population models (CGPMs), a computational abstraction that extends directed graphical models and can be used to describe and compose a broad class of probabilistic data analysis techniques. Examples include discriminative machine learning, hierarchical Bayesian models, multivariate kernel methods, clustering algorithms, and arbitrary probabilistic programs. We demonstrate the integration of CGPMs into BayesDB, a probabilistic programming platform that can express data analysis tasks using a modeling definition language and structured query language. The practical value is illustrated in two ways. First, the paper describes an analysis on a database of Earth satellites, which identifies records that probably violate Kepler's Third Law by composing causal probabilistic programs with nonparametric Bayes in 50 lines of probabilistic code. Second, it reports the lines of code and accuracy of CGPMs compared with baseline solutions from standard machine learning libraries.
The authors introduce conditional generative probabilistic models (CGPMs), which are essentially probabilistic models with a partially specified conditional dependence structure. A CPGM can embed submodels for which the conditional dependence structure need not be known, as along as it is possible to sample and evaluate the density for any conditional distribution on a subset of leaf node variables in the model. CGPMs can be composed into larger CGPMs, for which approximate implementations for the sampling and density functions can be defined using an importance sampling method. The authors present case studies of analysis performed using CGPMs that are implemented with BayesDB and Venturescript.  
I think this is a strong paper. The authors on the one hand define a clear mathematical abstraction, the CPGM, and on the other hand show case studies of analysis performed with a probabilistic programming system that implements this abstraction. The CPGM is described in reasonably precise mathematical notation, which formalizes the aspects of this work that generalize beyond the specific implementation. The cases studies are not easy to follow without familiarity with the BayesDB system, but do a good job of illustrating the general idea to this approach, which is to provide a high level modeling/inference abstraction in which black box components can be embedded. The paper is reasonably clear. It is fairly heavy on notation, but most aspects of the presentation can be understood on a sufficiently close reading. I have pointed out point where presentation can be improved below.   Questions and Suggestions  - It would be helpful to explicitly define what is meant by "query" and "evidence" in the context of this paper, which currently has to be inferred by the reader from the simulate / logpdf interface.   - What is the distinction between the observed output nodes D and the evidence node x_{r,E}? Does D refer values {x_r: r \in D} for which *all* variables are *always* observed?  - In figure 1, why is there a plate around x and not around z? I would avoid using the the index i when writing {z_i : i \not= r} in the caption, since the authors use x_[r,i] in the next sentence.  - I was unable to find a definition for {q_j} in the simulate interface. Presumably q_j \in [T]? Likewise the notation x_[r,Q] presumably refers to {x_[r,q] : q \in Q}?   - What is meant by "the subset of the outputs x^a_[*, A]"? In particular, what does the "*" notation mean?  - In algorithm 1, please define explicitly which symbol is bound to which input in the functions simulate and logpdf. Having to infer this from context makes it not obvious what is going on.   - I'm confused by the simulate implementation. Line 3 generates a sample conditioned on both the evidence and the queried variables, does it not? So then, why would {x_[r,Q^k] \in s^m}?, since in line 24 we simulate \E^k? Should line 3 perhaps read      Weighted-Sample(x[r,E_k])  - This is a matter of taste, but I would prefer for the authors to not use 'w' to describe a log weight, since most of the importance sampling literature uses 'w' to refer to an importance weight (i.e. the exponent of the log weight w used by the authors).   - evdience -> evidence (please run a spell check)