Among multiple ways of interpreting a machine learning model, measuring the importance of a set of features tied to a prediction is probably one of the most intuitive way to explain a model. In this paper, we establish the link between a set of features to a prediction with a new evaluation criterion, robustness analysis, which measures the minimum distortion distance of adversarial perturbation. By measuring the tolerance level for an adversarial attack, we can extract a set of features that provides the most robust support for a current prediction, and also can extract a set of features that contrasts the current prediction to a target class by setting a targeted adversarial attack. By applying this methodology to various prediction tasks across multiple domains, we observe the derived explanations are indeed capturing the significant feature set qualitatively and quantitatively.
The manuscript proposes a method for model explanation and two metrics for the evaluation of methods for model explanation based on robustness analysis. More specifically, two complementary, yet very related, criteria are proposed: i) robustness to perturbations on irrelevant features and ii) robustness to perturbations in relevant features. Moreover, different from existing works which defined the perturbation values following different somewhat-fixed procedures, the proposed method aims at allowing perturbations in any directions.
A greedy algorithm optimizing these criteria is proposed in order to produce a method able to highlight important features of the input and justify/explain model predictions.
In addition, the proposed robustness criteria are used as metrics to assess the performance of methods for model explanation.
Experiments on models addressing image classification and text classification, shows the performance of the proposed method w.r.t. existing work.
The manuscript has a good flow, and its content is easy to follow. The proposed method is sound, well motivated, and very well founded. The formal presentation of the proposed method is good. I appreciate the fact that evaluation covers different modalities of data, i.e. text and images.
My main criticism over the manuscript is the following:
In Sec. 3.3, when describing the first criterion, it is stated that the size |S_r|, i.e, the amount of anchors, could be defined by the used. In my opinion, this may not be applicable since in theory the amount of relevant/irrelevant features is unknown before hand. In that case the proposed AUC-based method seems more adequate. Could you comment on this?
In  Sec. 3, a pre-defined size K is introduced. Later in Sec. 3.1 it is stated that the greedy algorithm uses this size as a stopping criterion for the optimization of the proposed robustness criteria. Could you indicate how this size K is defined in practice? Is there a principled way to define it? What is the effect of this parameter on the performance of the proposed method? An ablation study focused on this parameter would provide further insights into the inner workings of the proposed method and would improve the manuscript.
In Sec.4 it is stated that only 50 random examples are considered when reporting results. When comparing the performance across the different methods, are these random 50 examples fixed or always re-sampled? Also, given the size of the considered datasets, where the number of images in their test sets is in the order of the thousands, it is hard to grasp how representative are the reported results?
In the same paragraph discussed above, it is mentioned that the GRAD method performs competitively on the proposed criteria. It might be interesting to further positioning the proposed method w.r.t. GRAD. Given the comparable performance achieved by GRAD and its relative simplicity, it would be hard to motivate why not choose GRAD instead of the proposed method? Could you provide some discussion on this?
In Sec.4 (pag. 6) it is stated the the proposed regression-greedy method outperforms other methods in these criteria. In my opinion this trend shouldn't be surprising given the fact that the proposed method is specifically optimized on such criteria as it is clearly stated by the title of Sec.3.
Fig.3 and Fig.4 display binary images indicating the top-n features selected by different methods. Perhaps it would be more informative to have a heatmap highlighting/grading the entire input space. This may throw more light on the performance of the compared methods.
In Adebayo et al., NIPS'18 (and very related efforts), there are presented a set of sanity checks to be applied to explanation methods to ensure their predictions are relate to the class and model being predicted. Could you provide any indication on whether the proposed method passes these checks?
In Sec.4 (visualization) it is stated that the proposed method effectively highlights crucial positive pixels as well as pertinent negative pixels. A similar capability has also being reported earlier in Samek et al., Trans NNLS'16 and Oramas et al. ICLR'19. Since the visualization analysis (discussed in pag.6) focuses exclusively on this capability. There should be a comparison between the proposed method and the two mentioned works.