Nonparametric learning from Bayesian models with randomized objective functions
Bayesian learning is built on an assumption that the model space contains a true reflection of the data generating mechanism. This assumption is problematic, particularly in complex data environments. Here we present a Bayesian nonparametric approach to learning that makes use of statistical models, but does not assume that the model is true. Our approach has provably better properties than using a parametric model and admits a Monte Carlo sampling scheme that can afford massive scalability on modern computer architectures. The model-based aspect of learning is particularly attractive for regularizing nonparametric inference when the sample size is small, and also for correcting approximate approaches such as variational Bayes (VB). We demonstrate the approach on a number of examples including VB classifiers and Bayesian random forests.
The idea: You want to do Bayesian inference on a parameter theta, with prior pi(theta) and parametric likelihood f_theta, but you're not sure if the likelihood is correctly specified. So put a nonparametric prior on the sampling distribution: a mixture of Dirichlet processes centered at f_theta with mixing distribution pi(theta). The concentration parameter of the DP provides a sliding scale between vanilla Bayesian inference (total confidence in the parametric model) and Bayesian bootstrap (no confidence at all, use the empirical distribution).  This is a simple idea, but the paper presents it lucidly and compellingly, beginning with a diverse list of potential applications: the method may be viewed as regularization of a nonparametric Bayesian model towards a parametric one; as robustification of a parametric Bayesian model to misspecification; as a means of correcting a variational approximation; or as nonparametric decision theory, when the log-likelihood is swapped out for an arbitrary utility function.  As for implementation, the procedure requires (1) sampling from the parametric Bayesian posterior distribution and (2) performing a p-dimensional maximization, where p is the dimension of theta. (1) is what you need to do in parametric Bayesian inference anyway, and (2) will often be a convex optimization problem, provided that your log-likelihood or your utility function is concave. You have to do (1) and (2) a bunch of times, but it's parallelizable.  The illustrations of the method are wide-ranging and thoughtful. Section 3.2 shows that the method can be used to correct a variational Bayes logistic regression model; the corrected VB samples look like MCMC samples from the actual Bayesian posterior but are faster to generate than the MCMC samples. Section 3.3 explores how posterior predictive samples from this method could be passed as pseudo-data to an outside analyst who is not permitted to access the data directly. This is an interesting idea, though I'm not sure how it would compare to simple noise addition (or whatever other methods are used in the privacy literature).  A quibble with the wording of lines 112-113 ("unlike MCMC, there are no issues or need for checking convergence..."), as the method does require sampling from the parametric Bayesian posterior, and this will often require checking convergence and using a burn-in phase, etc.  In practice I think I'd struggle with choosing c, the DP concentration parameter, but when doing parametric Bayesian inference (c=infinity), it could be good practice to redo the inference under various finite values of c as a form of sensitivity analysis. The nice thing is that you can use the same set of posterior draws across all values of c, which is a real time-saver.  In summary, this is a well-written paper that presents several potential applications of a simple, good idea. I recommend acceptance.