Interpretability methods often measure the contribution of an input feature to an image classifier's decisions by heuristically removing it via e.g. blurring, adding noise, or graying out, which often produce unrealistic, out-of-samples. Instead, we propose to integrate a generative inpainter into three representative attribution methods to remove an input feature. Compared to the original counterparts, our methods (1) generate more plausible counterfactual samples under the true data generating process; (2) are more robust to hyperparameter settings; and (3) localize objects more accurately. Our findings were consistent across both ImageNet and Places365 datasets and two different pairs of classifiers and inpainters.
The paper is focused on perturbation-based local explanation methods; methods that only need black-box(ish) access to the model and generally seek to find a region, pixel, etc's importance score through by removing that region, pixel, etc.. The intuition is that an important region if removed, will result in a large drop in the predicted class confidence. One main issue with such methods is that the removal itself can throw the image out of the data distribution and therefore result in a drop in confidence, not because of the region's importance, but because of the network's unpredictable behavior in such unseen parts of the input space. The work is focused on giving a solution to this problem: instead of removal through blurring, graying, etc, use inpainting; i.e. replace the removed region with using given the rest of the image. The idea has already been discussed out in the literature and the novelty of the work seems to be twofold: They introduce the same method in a way that is not curated for a specific perturbation-based method and could be concatenated with "ANY" given (or future) perturbation-based local explanation method (which authors notate by calling it ${existing_method}-G, they study robustness to hyper-parameter choice. 
The paper is quite well written and the experiments are comprehensive. I have two major comments/issues with the work:
1- The contribution of this work given existing work (more specifically the famous Chang et al work seems not to be enough for a venue like ICLR. If I want to list the contributions, it would be as follows (I would appreciate if the authors could correct me as the score is subject to change given more clarification on the matter):
    - This work utilizes an inpainting step in combination with several methods while previous work is focused on meaningful perturbations method. This, although useful, does not introduce a novel technical contribution. The main technical contribution has been the use of inpainting (to be more exact, using generative models to approximate P(c|x_r)) which has been done before on a few previous works.
    - The work argues that the use of inpainting in Chang et al (focused on keeping the salient object and removing background) was invalid as the inpainter model is not trained to do such a task. It could be argued that one could train another inpainting model that "is" capable of such a thing and therefore the general argument would not hold. One drawback of this approach, however, would be that training such an inpainting model might be difficult.
    - Hyperparameter robustness. Studying this question is valuable. However, given that the assumption of this work and previous works is that generative approaches are generally better (even not considering the hyperparameter robustness), I am not sure how this knowledge could be used.
2- Both this work and the previous works run on the assumption mentioned at the beginning of this review which basically says that non-generative perturbation-based methods throw the image out of data distribution and this is bad for such and such reasons. Although intuitively clear, I could not find any evidence in this work suggesting any meaningful difference using objective measures. One would assume that such phenomena would manifest itself clearly using insertion-deletion explanation metrics while as the authors report there was no significant difference. (Section 4.1 results clearly show a difference but this is not related to how the downstream explanation task is affected) For all it is know, generative methods have the drawback of being computationally more expensive than a simple blurring or replacing with random noise. (And a major elephant in the room is whether using an inpainter is actually taking the data back to the true data distribution which seems to be on an unproven assumption that these generative models are capable of learning the data manifold)
Minor comments:
    -Section 4.2 is really interesting. Thanks!
    - Fig 3 results: MP is more robust than MP-G and I couldn't find any explanation of why this method behaves specifically different than the other two in the experiments section. It might be better to move the explanation in the discussions to the experiments.
    - The task of most generative perturbation-based methods is to find a way to approximate P(c|x_\r) which is the conditional probability given the non-removed part of the image. Usually, they do the approximation by sampling several images from the conditional P(x_r|x_\r) (conditional inpainting)  using the generative model and averaging the prediction probability. This work seems not to be concerned with these specifics and directly feeds one of such samples. Could you explain this choice
    - For studying the robustness of LIME, apart from the random seed, couldn't one change the hyperparameters of the superpixel method? Tha one seems more of a practical problem.