Provable Variational Inference for Constrained Log-Submodular Models
Submodular maximization problems appear in several areas of machine learning and data science, as many useful modelling concepts such as diversity and coverage satisfy this natural diminishing returns property. Because the data defining these functions, as well as the decisions made with the computed solutions, are subject to statistical noise and randomness, it is arguably necessary to go beyond computing a single approximate optimum and quantify its inherent uncertainty. To this end, we define a rich class of probabilistic models associated with constrained submodular maximization problems. These capture log-submodular dependencies of arbitrary order between the variables, but also satisfy hard combinatorial constraints. Namely, the variables are assumed to take on one of -possibly exponentially many -set of states, which form the bases of a matroid. To perform inference in these models we design novel variational inference algorithms, which carefully leverage the combinatorial and probabilistic properties of these objects. In addition to providing completely tractable and well-understood variational approximations, our approach results in the minimization of a convex upper bound on the log-partition function. The bound can be efficiently evaluated using greedy algorithms and optimized using any first-order method. Moreover, for the case of facility location and weighted coverage functions, we prove the first constant factor guarantee in this setting -an efficiently certifiable e/(e − 1) approximation of the log-partition function. Finally, we empirically demonstrate the effectiveness of our approach on several instances.
The authors present an algorithm for approximate inference in exponential family models over the bases of a given matroid.  In particular, the authors show how to leverage standard variational methods to yield a provable approximation to the log partition function of certain restricted families.  This is of interest as 1) these families can be difficult to handle using standard probabilisitic modeling approaches and 2) previous bounds derived from variational methods do not necessarily come with performance guarantees.  General comments:  Although the approach herein would be considered variational approximation methods, the term variational inference is more commonly used for a related but different optimization problem.  There are lots of other variational approaches that yield provable upper/lower bounds on the partition function.  Some effort should be made to cite these.  A few to help the authors get started (see citations therein for more):  Tamir Hazan and Amnon Shashua. "Norm-product belief propagation: Primal-dual message-passing for approximate inference." IEEE Transactions on Information Theory 56, no. 12 (2010): 6294-6316.  Meltzer, T., Globerson, A. and Weiss, Y., 2009, June. Convergent message passing algorithms: a unifying view. In Proceedings of the twenty-fifth conference on uncertainty in artificial intelligence (pp. 393-401). AUAI Press.  Nicholas Ruozzi. Beyond log-supermodularity: lower bounds and the Bethe partition function. Uncertainty in Artificial Intelligence (UAI), July 2013.   Nicholas Ruozzi.  The Bethe partition function of log-supermodular graphical models. Advances in Neural Information Processing Systems (NIPS), December 2012.  Typos: Line 222, "which is easily seen to concave" -> "which is easily seen to be concave"