Interpretable machine learning has gained much attention recently. Briefness and comprehensiveness are necessary in order to provide a large amount of information concisely when explaining a black-box decision system. However, existing interpretable machine learning methods fail to consider briefness and comprehensiveness simultaneously, leading to redundant explanations. We propose the variational information bottleneck for interpretation, VIBI, a system-agnostic interpretable method that provides a brief but comprehensive explanation. VIBI adopts an information theoretic principle, information bottleneck principle, as a criterion for finding such explanations. For each instance, VIBI selects key features that are maximally compressed about an input (briefness), and informative about a decision made by a black-box system on that input (comprehensive). We evaluate VIBI on three datasets and compare with state-of-the-art interpretable machine learning methods in terms of both interpretability and fidelity evaluated by human and quantitative metrics.
[Due to the rebuttal, my score was raised from a weak reject to a weak accept]
Summary
The paper addresses the problem of interpreting predictions/decisions of a black-box classifier/regressor by masking the parts of the input that were most relevant. The proposed approach consists of, first, manually designing “cognitive chunks” of input data, e.g. individual words for sentiment classification or fixed-size image-patches for image-classification. Then, a variational IB framework is used to infer which of these chunks are relevant for the classifier’s decision. Additionally, there is a (hard) constraint, making sure that only a fixed (small) number of chunks is used. The bottleneck variable, in this case, is a sparse-chunk representation of the data. The latter is obviously a more compressed representation of the data, but importantly it is a more compressed representation that contains the largest possible amount of relevant information about the decision (because of properties of the information-bottleneck objective). Both factors together, according to the paper, constitute a “good” (i.e. brief but comprehensive) explanation which allows for interpretability and attribution of the black-box system’s decision. The method is evaluated on three tasks (sentiment prediction, image classification, TCR to epitope binding prediction) and performance is reported to be on-par or better than state-of-the-art methods.
Contributions
-) Application of the IB-method for generating summaries of decision-relevant input-data, which are good candidates for interpretability. The theoretical properties of the IB objective are appealing for producing interpretable data-summaries.
-) Adaptation of the variational IB framework, using bits and pieces reported in the literature such that the bottleneck variable is a sparse, binary vector over “cognitive chunks”.
-) Experimental evaluation, where human judges rate the “interpretability“ of various state-of-the-art attribution methods.
Quality, Clarity, Novelty, Impact
The paper addresses a timely and important problem, particularly the IB framework could add some solid theoretical footing (the “theory of relevant information”) to the field of interpetability methods. The paper is well written (though it needs another pass for typos, etc.), related methods and literature are discussed and compared against, and the specific variational IB objective is introduced nicely. Large parts of the method (deep variational IB, VI with categorical variables) have been published before, but these parts are combined in a novel and original way. My main issues with the current paper are (I) interpretability and comprehensiveness are not necessarily the same as maximum compression of maximally relevant information, (II) the method (in theory) depends strongly on the quality of the approximator, this is currently not mentioned and not explored, (III) the experimental section is currently not very strong, in particular the MNIST experiment. See more details for the main issues below. Overall, I personally think that the main idea of the paper is interesting, mature and fleshed out enough for a publication, however, the experimental section is somewhat lacking and (II) is missing from the current manuscript. While the method has theoretical advantages, empirically it seems to perform more or less equal to L2X (but the chunks produced seem qualitatively different which is interesting). I am therefore slightly leaning towards suggesting a major revision of the paper, but I am happy to be convinced otherwise by the other reviewers and the authors during discussion/rebuttal. (I would rate the paper as "borderline", but it seems that this year's review system only allows for "weak reject" or "weak accept", so I'll go for "weak reject" for now).
Improvements / major issues
(I) Good compression of highly relevant information is not (always) the same as good interpretability/comprehensiveness. In the limit, the bottleneck variable captures a minimal sufficient statistic, i.e. a maximally compressed version of all relevant information - for finite beta, the bottleneck approximates such a minimal sufficient statistic. From a theoretical point of view this is very appealing, since it is guaranteed to cover a maximal amount of information (given a certain level of compression). But the way this information is represented matters a lot for interpretability - any reversible mapping of the bottleneck variable does not change its information content but can have substantial effects on interpretability, e.g. consider encrypting or randomly perturbing elements of the explanation (i.e. the selected cognitive chunks). This is a major open problem, and some theoretical grounding in the IB framework helps by talking about this problem in very concrete terms. While I would not expect the paper to solve the problem in full generality, some discussion, and perhaps adding a “shortcomings” section would be nice.
(II) The relevance of information is measured via I(t;y), which ultimately boils down to the approximation q(y|t). The quality of this approximation is crucial, which can of course be seen by how it influences the tightness of the bound. While I appreciate that the paper investigates the quality of the approximation to some degree (by inspecting the approximator fidelity), I would highly appreciate a thorough discussion of this issue (because ultimately the method will produce cognitive chunks that are relevant for q(y|t), not p(y|x) - the interpretations can be trusted only if q matches fairly well). It would be very interesting to see how quickly interpretability degrades with lower-quality q(y|t) - the latter would of course require more experiments with human “interpreters” which I would not expect to be easily feasible within the rebuttal period.
Another interesting experiment to test the match between q(y|t) and p(y|x) would be to “minimally intervene” on the input-chunks suggested by the method and see whether that actually affects the predictions of the black-box models. E.g. do small random perturbations to the selected cognitive chunks in the MNIST digits change the prediction of the black-box classifier? Compare this against small random perturbations in arbitrary chunks of the input.
(III) Experimental section: I’m fairly happy with the IMDB experiment, and the TCR to Epitope binding is a nice non-standard application but I find the quality and significance of the results a bit hard to judge. My main concern though is the MNIST experiment: what I would have expected was the following: cognitive chunks are shown to participants and they need to guess the correct number (just like in the IMDB experiment). In the experiment reported in the paper, I’m afraid that there’s a certain bias for judges favoring explanations that lie on the digits rather than off digits. It remains unclear whether they simply prefer the chunks selected by VIBI over other methods, or whether they have actually gained more understanding of how the black-box makes decisions.
(IV) Table 2 can easily be misleading because entries with highest mean-accuracy are marked in bold, regardless of whether confidence intervals overlap with other entries or not. Please fix this by either only marking entries in bold where the error bars don’t overlap with an entry in the same row, or marking all entries in bold that lie within the error bars of the best-performing entry. Particularly for “Approximate Fidelity” VIBI often does not perform significantly better than L2X but performs roughly equally well. Of course it would also help to run more repetitions to potentially shrink confidence intervals.
(V) Please state the (parametric form) of the prior r(z*) used for the experiments. Also state the analytical expression for the KL-term in the final objective that this prior leads to.
Minor comments
a) Please add some discussion on how the method depends on hand-crafting cognitive chunks, and how hard/easy this might be for different domains.
b) Rather than fixing r(z), other papers have proposed to optimize the prior as well (typically in the context of VAEs / VIB) which is well justified from an IB perspective. It might be interesting to explore these possibilities for VIBI as well in the future.
[1] Fixing a broken ELBO. Alemi et al. 2017
[2] The beta-VAE’s Implicit Prior. Hoffman et al. 2017
c) Instead of fixing the number of cognitive chunks in advance, it could also be interesting to infer that number as well (as a future extension of the method). This could either be achieved via a sparsity-inducing prior r(z), or perhaps by borrowing some ideas from the Deterministic IB [3], and its variational version.
[3] The deterministic information bottleneck. Strouse and Schwab. 2016