Modularity is a key feature of deep learning libraries but has not been fully exploited for probabilistic programming. We propose to improve modularity of probabilistic programming language by offering not only plain probabilistic distributions but also sophisticated probabilistic model such as Bayesian nonparametric models as fundamental building blocks. We demonstrate this idea by presenting a modular probabilistic programming language MXFusion, which includes a new type of re-usable building blocks, called probabilistic modules. A probabilistic module consists of a set of random variables with associated probabilistic distributions and dedicated inference methods. Under the framework of variational inference, the pre-specified inference methods of individual probabilistic modules can be transparently used for inference of the whole probabilistic model. We demonstrate the power and convenience of probabilistic modules in MXFusion with various examples of Gaussian process models, which are evaluated with experiments on real data.
In this paper authors present a new Probabilistic Programming Language (PPL) MXFusion. Similarly to the languages for the deep learning (TensorFlow, PyTorch, etc.), this language introduce probabilistic modules that are used as building blocks for complex probabilistic models. Introducing modularity to the probabilistic programming, raises the problem of inference for probabilistic models. Since, we cannot obtain the exact solution on practice we have to resort to approximate inference methods. The approximate inference methods can be either generic, thus, being suitable for many probabilistic models but resulting in poor approximation, or specific, thus, having good approximation quality, but only for specific probabilistic models. Authors propose to address this problem by encapsulating specific inference methods in corresponding probabilistic modules. Doing so, one can perform approximate inference for every module with the best suitable inference technique. Authors demonstrate interface of MXFusion for three well known probabilistic models: Bayesian linear regression, deep kernel learning, Bayesian Gaussian process latent variable model.
Approaching the problem of building complex probabilistic models by introducing modular PPL is an important direction of study. But, regarding this paper I have the following concerns.
- In my opinion, the structure of the paper can be greatly improved. From general words about modularity and approximate inference authors dive to the very specific cases of probabilistic models. Following such structure, authors don’t give a clear answer to the following questions. Why the paradigm of encapsulating inference methods in probabilistic modules is legitimate for constructing complex probabilistic models? What inference methods and probabilistic models can we use as building blocks? Do we need to be aware of specific inference methods that are encapsulated or we can use any blocks in any order as we do in deep learning frameworks?
- Novelty of that paper is the new design of PPL. That is an interesting and important question for the community, but maybe ICLR paper is not the best format to present such kind of novelty. 
- From the specific examples in the paper, legitimacy of such modular structure is clear only for variational inference (that seems to be a common knowledge) and variational approximation of gaussian processes. But the application area of MXFusion remains unclear. Verbatim examples of code for the specific examples doesn’t make the difference between MXFusion and other PPLs clear, because it can be treated as encapsulation of the code into some classes, that can be implemented in other languages as well.
- Comparison with other frameworks can be improved. In experimental section authors provide comparison with GPy framework in terms of RMSE and log-likelihood for gaussian process with 50 inducing points. As I understood both frameworks use the same inference methods and achieve the same performance, so the experiment can be considered as sanity check for MXFusion. The paper could benefit from comparison between different inference methods and providing benchmarks for inference time.
Overall, the paper proposes a new PPL that is an important direction of study, but have several drawbacks and conference paper format is not the best way to present such kind of novelty.
Typos:
- Page 1, “despite the different of DNNs…” -> “despite the difference of DNNs…”?
- Page 2, missing reference of the section
- Page 2, section 3, “... sightly different form.” -> “... slightly different form”?