The problem of explaining deep learning models, and model predictions generally, has attracted intensive interest recently. Many successful approaches forgo global approximations in order to provide more faithful local interpretations of the model's behavior. LIME develops multiple interpretable models, each approximating a large neural network on a small region of the data manifold, and SP-LIME aggregates the local models to form a global interpretation. Extending this line of research, we propose a simple yet effective method, NormLIME, for aggregating local models into global and class-specific interpretations. A human user study strongly favored class-specific interpretations created by Norm-LIME to other feature importance metrics. Numerical experiments employing Keep And Retrain (KAR) based feature ablation across various baselines (Random, Gradient-based, LIME, SHAP) confirms NormLIME's effectiveness for recognizing important features.
This paper proposes a novel scheme to aggregate local explanations
from LIME to generate global and/or class-level interpretations in the
form of feature importances. The technical novelty lies in the
normalization of the feature weights across local explanations where
the proposed scheme utilizes a well-motivated L0 normalization in
place of the existing L2 normalization in Ribeiro et al., 2016. The
sparsity induced by the L0 normalization leads to more "easy-to-read"
global (or class level) "explanations" since the number of features
with non-zero importances is reduced.
The simple idea of generating sparse aggregations of local
explanations to obtain global (or class-level) interpretations is well
motivated and seems fairly intuitive (and I consider simplicity a
strength) and the extremely favourable empirical results, I am leaning
towards a reject for the current version of the paper. The main
reasons for this decision are as follows: 
- The proposed scheme is presented as closely tied to LIME, which
  severely limits the scope of the proposed solution. However, I can
  imagine that the high level idea of sparse aggregation of local
  explanations can easily generalize to many local explanation
  schemes. 
- The empirical evaluation needs to be improved (or better clarified)
  in my opinion. There are many potential baselines that are also not
  considered (see details below). Moreover, the evaluation is
  completely limited to a single data set (which is good as a strawman
  but not sufficient to make significant claims).
Clarifications re: baselines and empirical evaluations:
- No comparison to Ibrahim, et al., 2019 -- you can cluster in
  unsupervised setting but you can also aggregate within class in
  supervised setting. Generating class-level interpretations is an
  interesting idea but its novelty is somewhat unclear.
- In terms of explanations, there are contrastive explanations [1]
  which are not really discussed in this paper. This scheme could be
  more useful for generating class level interpretations.
- It is not clear why the negative correlations were removed in the
  user study. In certain situations, such as when distinguishing
  between digits 3 & 8, the absence of weights (potentially
  corresponding to negative weights) can be crucial explanations.
- It is not clear why in the SmoothGrad and VarGrad baselines, the
  class level interpretations are generated with just 10 images per
  class. Are the class level interpretations of LIME and NormLIME also
  generated from just 10 images per class. If that is the case, 10
  seems like a very small number and can potentially create noisy
  explanations. If LIME and NormLIME used more than 10 images to
  generate class level interpretations, the comparison does not seem
  fair. Please explain this discrepancy.
- For the evaluation in Sec. 5, it is not clear how the features are
  removed. Are they removed based on the global explanations (feature
  importances) of each baseline? If this is the case, are the global
  explanations of each baseline generated using the same amount of
  local explanations? Please clarify.
- The KAR analysis results in Fig. 4(b) need to be further
  investigated. First, it is not clear if this difference between the
  different baselines something that happens in multiple datasets or
  just this one. Moreover, it seems natural that something like
  SmoothGrad and VarGrad would be able to capture the feature
  redundancies if the redundancies were the actual cause for this
  marked difference between the relative performances in Fig. 4(a) and
  4(b). This could definitely use a better discussion.
[1] Dhurandhar, Amit, et al. "Explanations based on the missing:
Towards contrastive explanations with pertinent negatives." Advances
in Neural Information Processing Systems. 2018.