Global Sensitivity Analysis for MAP Inference in Graphical Models
We study the sensitivity of a MAP configuration of a discrete probabilistic graphical model with respect to perturbations of its parameters. These perturbations are global, in the sense that simultaneous perturbations of all the parameters (or any chosen subset of them) are allowed. Our main contribution is an exact algorithm that can check whether the MAP configuration is robust with respect to given perturbations. Its complexity is essentially the same as that of obtaining the MAP configuration itself, so it can be promptly used with minimal effort. We use our algorithm to identify the largest global perturbation that does not induce a change in the MAP configuration, and we successfully apply this robustness measure in two practical scenarios: the prediction of facial action units with posed images and the classification of multiple real public data sets. A strong correlation between the proposed robustness measure and accuracy is verified in both scenarios.
The paper gives a nice introduction to sensitivity analysis in graphical models, and proposes a method to check if the MAP configuration will not change, if a simultaneous perturbation in the parameters occurs. The method works by selecting perturbed factors in such a way (using local computations) to create a worst-case scenario. The second best MAP estimation is then identified for the worst case scenario. Theoretically, the complexity is therefore equivalent to the MAP query (exponential in the tree-width), if the number of variables is a constant. As presented, one disadvantage of this technique is that, practically, it requires computational time: n*[MAP inference time], where n is the number of variables. For non-trivial graphical models, the proposed sensitivity analysis can be orders of magnitude slower than the already hard MAP query. That said, more efficient algorithms for getting the M-Best MAP seem to exist and the authors may consider to utilize one of them as a black box. See for example: An Eﬃcient Message-Passing Algorithm for the M-Best MAP Problem, by Dhruv Batra. See also: Search Algorithms for m Best Solutions for Graphical Models, by Rina Dechter et al. On the other hand, the experimental results in the paper do not provide very useful information, as discussed below.Quality:The paper appears to be technically correct. Theorem 1 is trivial and could be a lemma for Theorem 2.  The proof of Theorem 2 in the supplementary material could be re-written in a clearer way to provide more insight. For example, you could discuss the choice of factors in the new PGM as creating a worst-case, and briefly mention that the partition function cancels out and has no effect.On the other hand, the experimental results show that the robustness measure is a good indication for the testing accuracy. While I see why there is a relation between robustness and testing accuracy, I think the relation is much more involved and may be linked to the bias-variance trade-off. Thus, the experiments seem to be distantly related to the current work and not the best choice, or perhaps not well-motivated. I have to admit though that it might be hard to design very meaningful experimental results in this case. The authors could perhaps verify the proof experimentally and show how pessimistic the analysis is, in real-world applications, by having an experimental measure of sensitivity.Clarity:The paper is clearly written when it talks about preliminaries, and related work. When it comes to the contribution, it lacks insightful explanation. Originality:To the best of my knowledge, the contribution is new.Significance:The paper tackles an important problem. Most of the technical results in the paper do not appear to be so surprising though. Worst-case analysis is widely used in many fields. If a system passes the worse case, it passes all cases.  While the paper proposed a novel exact method to do such analysis, in the context of MAP estimation, the inefficiency of the method, and the lack of meaningful experimental results have negatively affected the quality of the work.On the other hand, in some cases, the proposed analysis may be too pessimistic. It might be interesting to try to give a probabilistic guarantee that is more efficient to compute.Other comments/suggestions:In Figure 1 (a): The font is too small.In Figure 1 (b): I suggest adding labels for the axes, and on the plot (maximum, minimum, ...etc).Equation 5: Did you forget to mention that x(1) is the best MAP?UPDATE: *******After reading the authors' response, I am convinced that part of my criticism may have been a bit philosophical, and, therefore, should not lead to rejecting the paper. I am glad to raise my score, and hope the authors can get their final version in the best form, if the paper is accepted.