From MAP to Marginals: Variational Inference in Bayesian Submodular Models
Submodular optimization has found many applications in machine learning and beyond. We carry out the first systematic investigation of inference in probabilistic models defined through submodular functions, generalizing regular pairwise MRFs and Determinantal Point Processes. In particular, we present L-FIELD, a variational approach to general log-submodular and log-supermodular distributions based on sub-and supergradients. We obtain both lower and upper bounds on the log-partition function, which enables us to compute probability intervals for marginals, conditionals and marginal likelihoods. We also obtain fully factorized approximate posteriors, at the same computational cost as ordinary submodular optimization. Our framework results in convex problems for optimizing over differentials of submodular functions, which we show how to optimally solve. We provide theoretical guarantees of the approximation quality with respect to the curvature of the function. We further establish natural relations between our variational approach and the classical mean-field method. Lastly, we empirically demonstrate the accuracy of our inference scheme on several submodular models.
This paper investigates an interesting idea of approximating the normalization constant (thereby computing various inference quantities like marginals, conditional distributions etc.) for the general class of log-submodular and log-supermodular distributions.On the positive side, I thought the idea of this paper was neat -- to use the semigradients to provide upper/lower bounds of the functions, and compute the constants for these bounds. In this way, they leveraged a lot of the existing results concerning approximation bounds, to measure the tightness of the approximation.A main drawback of this paper is that the results seem theoretically pretty weak, given that the main contribution of this paper is theoretical. Theorem 2 is, in my opinion, the only result which talks about the tightness of the approximation, and provides a factor, which is in the worst case O(n). I wonder if there is a notion of hardness factor in computing the normalization factor (using, for instance, the construction of [Balcan and Harvey, 2009]). In my opinion, the paper is somewhat incomplete without a discussion on this, or even possible directions of computing the normalization factor. Without a notion of hardness, the worst case factor can look very discouraging, and there is no sense of how hard this problem can be (apart from the #P-hardness result, which is stated in lines 150 -- it would be nice to have a reference or a proof of this). Can one say something like, no polynomial time algorithm can compute Z up to a factor of p(n), where p(n) is a polynomial in n?I would also have loved to see some more discussion on the utility of computing these quantities. For instance, does this give method to approximately sample from a log-submodular distribution? Does this give guarantees in terms of learning mixtures of submodular functions? It seems to me that it would not be hard to show guarantees for Maximum-Likelihood learning of mixtures of submodular functions under this model. Consider the following setup. We have a mixture of submodular functions, and define p(X) \propto \exp(\sum_i w_i f_i(X)). One could then do approximate max-Likelihood learning to learn the weights w. This might be an interesting application of the framework. It would also be nice to see some stronger empirical utility of these models on real world problems, like for e.g. summarization. Does this framework admit sampling algorithms from submodular functions? On a whole, the paper misses some important aspects of this story. I would really love to see more discussion on this.