On Prior Distributions and Approximate Inference for Structured Variables
We present a general framework for constructing prior distributions with structured variables. The prior is defined as the information projection of a base distribution onto distributions supported on the constraint set of interest. In cases where this projection is intractable, we propose a family of parameterized approximations indexed by subsets of the domain. We further analyze the special case of sparse structure. While the optimal prior is intractable in general, we show that approximate inference using convex subsets is tractable, and is equivalent to maximizing a submodular function subject to cardinality constraints. As a result, inference using greedy forward selection provably achieves within a factor of (1-1/e) of the optimal objective value. Our work is motivated by the predictive modeling of high-dimensional functional neuroimaging data. For this task, we employ the Gaussian base distribution induced by local partial correlations and consider the design of priors to capture the domain knowledge of sparse support. Experimental results on simulated data and high dimensional neuroimaging data show the effectiveness of our approach in terms of support recovery and predictive accuracy.
This paper considers the problem of doing Bayesian inference when one would like to impose structural priors that restrict the support of the distribution.  It begins by motivating incorporating support constraints via the maximum entropy principle, where we would like to impose the constraint but deviate from some base distribution as little as possible, as measured by relative entropy. The distribution that optimizes this criteria is called the information projection of the base distribution onto the constraint set.  The paper begins by showing that this principle leads to intuitive results in the case of support restrictions (the information projection of p onto support set A is 1{x \in A} p(x) / \int_{x \in A} p(x) ), and that the notion can be used to describe Bayesian inference, where the posterior arises when the joint distribution is projected onto the support set defined by the observations.When we would like to perform an information projection onto the intersection of several sets, then the paper shows that this can be done by projecting onto each set in sequence in any order and the results are unchanged. This motivates an inference procedure in the special case of sparsity restrictions, which leads to the paper's main algorithm. The idea of the algorithm is to structure inference as an inside and outside optimization, where in the inner optimization, a posterior is computed where the set of k entries allowed to be nonzero are fixed, and in the outer optimization, the set of nonzero entries is optimized over. It is shown that this optimization problem can be formulated as a submodular optimization problem, which leads the authors to propose a greedy forward selection strategy for choosing nonzero dimensions. Results against a variety of baselines show the method to work quite well.The main pros of the paper are as follows:- the writing is very clear, and the arguments appear to be clearly justified; things seem technically correct- the formulation in terms of information projections is a clean and interesting way of formalizing things, providing arguments that help better understand what it means to add constraints to models- the algorithm appears to work wellThe main con is that there is a lot of technical work to get to results that are not terribly surprising. From a practical perspective, I'm not convinced that Section 2 of the paper is necessary to get to the algorithm in Section 3. It seems that we could have simply observed that the sparsity constraints can be viewed as a union of simple constraint sets, then arrived fairly naturally at the objective that is used in Section 3, and then derived the algorithm in the paper. The fact that J(s) is submodular is easy to see. So while I enjoyed reading the paper, I'm not sure I left with any new tools for imposing prior knowledge in my models.