Existing local explanation methods provide an explanation for each decision of black-box classifiers, in the form of relevance scores of features according to their contributions. To obtain satisfying explainability, many methods introduce ad hoc constraints into the classification loss to regularize these relevance scores. In this paper, we discuss some shortcomings of these methods and address them by a simple but effective mask predictor. Specifically, we present the concept of distribution controllers and integrate it with a neural network to directly guide the distribution of relevance scores. Then we introduce the classification loss to optimize the proposed predictor. The benefit of this strategy is to enable discriminative scores over supporting features, and facilitate the setting of involved hyperparameters. The experimental results demonstrate that the proposed method also outperforms others in terms of faithfulness and explainability. The code is available at https://github.com/iclrlocal/.
Paper Summary:
This paper proposed a method to produce instance-wise saliency map for image classification tasks. The proposed method develop a U-net-based generator for saliency mask, where the important modifications are (1) a skewness-inducing activation function for mask generation (i.e. controller), which is either a ReLU function or a scaled sigmoid function raised to a certain power. Authors argue that the proposed activation function leads to output saliency score to have right-skew distribution, which leads to more distinguished saliency map. (2) a smoothness-inducing mechanism where the saliency mask is generated at coarser scale then up-sampled with a bilinear operation so the generated map is smooth. Finally, the training is performed by minimizing solely on the cross-entropy loss with respect to the original model predictions, which is expected to improve faithfulness in mimicking the original black-box classifiers.  
I'm leaning toward rejecting this paper in its current form. While I think this paper proposed an interesting strategy in improving the faithfulness of the explainer (i.e. training on the cross-entropy loss with respect to original classifier), the rest of the two modification either already exists in the literature (generate mapping at coarser resolution is an idea from Dabkowski & Gal (2017) / Du (2018) as pointed out by the author) or suffers potential technical issues that can benefit from further methodology improvement/empirical justification (please see Major Comments). Furthermore, empirical experiments seem to suggest that the proposed method sometimes fail to produce more smooth and more discriminative maps compared to its MGnet baseline (e.g. Figure 5, top right row), calling into the question of whether training solely on a cross-entropy loss is enough to ensure the generated saliency map is of high-quality. I still like the overall idea of this paper, and I think this work can be made more rigorous and informative by including a careful ablation study about the loss/gain of different regularization terms and loss functions (e.g. the effect of TV(M) term on smoothness, the effect of f(\phi(I, 1-M)) term on discriminativeness, and the trade-off between cross-entropy v.s. the original negative log likelihood loss as used in Dabkowski & Gal (2017)).
Major Comments:
(1) Custom Transformer
The derivation of the new activation function assumes that the distribution of the hidden-layer output $m^g_{ij}$ is strictly Gaussian (i.e. unimodal, symmetric, and with sufficiently light tail), and author proposed to build this Gaussian distribution from raw hidden-unit output through standardization (Section 3.2.2). However, this approach may not be valid in practice. This is because there is no guarantee that the distribution of $m^g_{ij}$ is strictly unimodal and symmetric. If the output is skewed/multimodal in anyway, the normalization operation won't produce a Gaussian distribution, since a linear transformation (i.e. normalization) of a skewed distribution is still a skewed distribution. As a result, I am worried that the hyperparameter derived in Section 3.2.3 may not always lead to right-skewed distribution as expected. If author would like to ensure the outcome distribution is right-skewed, it may be good to tune these activation function parameters for each instance so the empirical distribution of $m^d_{ij}$ satisfy certain skewness criteria (e.g., maybe tune the parameter in a way such that the pearson's coefficient of skewness of the empirical distribution is sufficiently small).
In addition, I find the author's claim about the benefit of right-skewed distribution (i.e. producing more discriminative maps) not yet well supported by the empirical evidence. For example, in Figure 4 the produced maps in the first three rows are relatively fuzzy, and the difference between the irrelevant features and the important features does not seem to be very high. The situation is only improved after the instance-wise finetuning. I think there are two possible reasons, either (a) the right-skewness of the actual distribution of the saliency map score in those images were not guaranteed (as mentioned earlier, I believe the current approach cannot guarantee the pre-activation units are Gaussian distributed, therefore the actual skewness of the post-activation values may vary depending on the input). (b) the right-skewness in the outcome distribution alone is not enough to guarantee good discrimininative behavior.
To clarify this (and provide more evidence for author's claim that skewness <-> better discrimination), author can consider visualizing the saliency map with varying degree of skewness (you can measure the skewness of the empirical distribution of the saliency-map score using pearson's coefficient of skewness), and show that the "discriminativeness" and skewness of the empirical distribution are correlated.
(2) Smoothness of the produced map
In the original MGnet work, the authors used two mechanisms to guarantee smoothness of the saliency map: (1) generate map at coarser level and (2) TV penalty on loss function. In this work author used only (1). As a result, the produced map in some cases (e.g. Figure 5, top right row for BF After, also Figure 7, left fifth row for BF after) appear jagged and not smooth. I'm wondering if adding TV penalty back would help somewhat. Again, a careful ablation study would be beneficial in clarifying this.
Minor Comments
(1) There's some minor technical issue in the description for Equation (6). Assuming $z_{ij}$ is Gaussian, it is $\eta$ AND sigmoid() COMBINED to "approach the uniform distribution", it is important to mention the sigmoid function since the uniform transformation used here is based on the Probability Integral Transform,   (i.e. if a random variable $z \sim \Phi(.)$ follows Gaussian distribution with CDF $\Phi(.)$, then $\Phi(z) \sim Unif(0, 1)$, i.e. you are transforming a Gaussian random variable using Gaussian CDF to get uniform distribution), and equation (6) is using $\eta$ AND sigmoid() combined together to approximate this Gaussian CDF [1].
(2) Equation (14) on page 13. There should be an $z^2$ term within the exponential function.
Reference:
[1] Gary R.Waissi, Donald F.Rossin (1996) A sigmoid approximation of the standard normal integral.