Simple MAP Inference via Low-Rank Relaxations
We focus on the problem of maximum a posteriori (MAP) inference in Markov random fields with binary variables and pairwise interactions. For this common subclass of inference tasks, we consider low-rank relaxations that interpolate between the discrete problem and its full-rank semidefinite relaxation. We develop new theoretical bounds studying the effect of rank, showing that as the rank grows, the relaxed objective increases but saturates, and that the fraction in objective value retained by the rounded discrete solution decreases. In practice, we show two algorithms for optimizing the low-rank objectives which are simple to implement, enjoy ties to the underlying theory, and outperform existing approaches on benchmark MAP inference tasks.
This paper studies the technique of "low-rank relaxations" for MAP inference.The idea is to solve the problem $max_{x\in +-1} x^T A x$ using a relaxed form $max_{X\in \R^{n\times k}} tr(X^T A X)$ where the second maximization is constrained so that rows of $X$ have unit length. The value of $k$ can be varied to get different kinds of approximation. Solutions to the second problem are turned into solutions to the first using "randomized rounding" in which the rows of $X$ are projected onto a random vector and rounded to plus or minus 1.This idea is not new but the present paper contributes theoretical bounds which guide the choice of $k$. The mathematics was interesting to me and I did not find any mistakes, although I did not check the proofs of any of the theorems. The paper's conclusion is that $k$ does not have to be very large to get good results.The relaxation-rounding ("R^3") technique was compared to Mean Field and Gibbs on synthetic data with good results. I especially liked the argument in Figure 2 to show that R^3 can avoid some local optima which trap Gibbs and MF.I also liked the emphasis on the flexible interface provided by the algorithm, allowing different treatment of sparse and dense weight matrices for example (Introduction, penultimate paragraph).I was not clear on how novel the R^3 algorithm is, apparently from section 6 paragraph 3, similar algorithms have been presented before, but with different emphasis and no theory. It would be good to be able to convey this earlier in the paper, and tell the reader explicitly if "R^3" is a new term that is being coined here...End of 4.3: "every variables"There is an argument in the appendix that we need only consider MRFs with pairwise interactions since an auxiliary variable can be introduced to simulate singleton factors. If the auxiliary variable has the wrong value in a solution, simply flip the sign of all variables, and the objective will be preserved due to sign-symmetry of the pairwise-only MRF. It would be nice to see this argument appear early in the paper since otherwise some readers won't realize that the algorithm is fully general.Quality: I think the paper is of high quality.Clarity: It is clear and I enjoyed reading it and learning some new mathematics.Originality: The paper is not terribly original, but seems to provide some important results on an algorithm of interest.Significance: I think it is reasonably significant. MAP is not my favorite problem, but it's useful in some areas, and studying it gives rise to interesting mathematics which may have wider applicability. I think the paper will be interesting to Bayesians as well as MAP people.