summaries: In this paper, the authors propose to explain the behavior of black-box neural networks by approximating a top-level hidden layer (no non-linearities afterwards, except possibly softmax) as a linear combination of the hidden layers obtained from different examples. These examples may come from the training data, but it is not required. The contribution of each input feature is also evaluated using an extension of integrated gradient across multiple dimensions. This paper introduces SimplEx, a method that approximate the hidden representation of a test example using the linear combination of hidden representations of examples in a pre-defined corpus. Further, the hidden representation of the test instance is associated with each input feature of these examples in the corpus. The paper demonstrates the quality of approximation of SimplEx (Sec 3.1, SimplEx outperforms kNN baselines) and demonstrate its practical utility in the experiment of predicting cancer mortality (Sec 3.2) by allowing model users to interpret model predictions and determine whether to trust the prediction. Detailed math derivation and additional experiments with MNIST and time series data are included in the appendix. This paper introduces SimplEx, which is a method to post-hoc explain a model prediction by computing from a user-defined set of examples, how the set of examples decompose the prediction. For example, the clinical risk prediction of a patient can be explained by a set of other patients and how much each patient influenced the prediction.   Contributions -	A post-hoc explanation method, called SimplEx, were users can choose the set of examples from which the explanation should be constructed. This is advantageous when the training set is not available or a user wants customized explanations, e.g. a doctor wanting to understand a predictions in terms of other patients she/he knows. -	SimplEx combines the ideas of feature saliency and example-based explanations. For each example, the method computes the percentage of this example contributing to explaining the prediction as well as which features of this example were important. -	A latent-space generalization of Integrated Gradients [10] -	Experimental evaluation that show SimplEx to be more precise and robust compared to baselines.  The main contribution of this paper is SimplEx: a method that aims to explain a test prediction by decomposing it into the weighted sum of nearest neighbors (from a user-specified corpus) in the latent space. They claim this method is more precise and robust than existing explanation-by-example-comparison methods (namely, KNN-like explanations) and distinguish this method from (often gradient-based) feature saliency methods. The paper builds off work that try to explain model predictions that use comparisons in latent variable space (e.g., Concept Activation Vectors, Deep KNN). SimplEx works PROVIDED the model has a hidden layer that linearly maps to the output (this is the layer that is used for explaining predictions).   - This constraint is significant for accurate weighting of examples. Note that if this is not satisfied by the model (say for instance a softmax is added right before the final output), it is possible to consider an earlier layer that satisfies the linear-mapping constraint as the output  Note too that if the latent space spanned by the user uploaded corpus does not contain the latent representation of the test prediction of interest (it likely will not), the test's latent is projected into the space. This means that it is possible to use this method on any input, regardless of the corpus. We can then measure how good this approximation is by calculating the residual between the corpus span and the test latent.  We can then "interpret the shift in latent space as resulting from a shift in the input space." This is the crux selling point of the paper, and it is accomplished using integrated (and projected) jacobians. This technique also lets us see the contribution of a feature from a corpus example.  The authors then proceed to (briefly) characterize their model on MNIST against dKNN inspired methods, and show a use case on a clinical risk dataset.  They claim the following novelty:  1. Freedom to choose comparison corpus, no need for this to equal the training set 2. The instance-level decompositions are valid in latent and output space, and are more robust than other explainability methods
abstractText: Modern machine learning models are complicated. Most of them rely on convoluted latent representations of their input to issue a prediction. To achieve greater transparency than a black-box that connects inputs to predictions, it is necessary to gain a deeper understanding of these latent representations. To that aim, we propose SimplEx: a user-centred method that provides example-based explanations with reference to a freely selected set of examples, called the corpus. SimplEx uses the corpus to improve the user’s understanding of the latent space with post-hoc explanations answering two questions: (1) Which corpus examples explain the prediction issued for a given test example? (2) What features of these corpus examples are relevant for the model to relate them to the test example? SimplEx provides an answer by reconstructing the test latent representation as a mixture of corpus latent representations. Further, we propose a novel approach, the Integrated Jacobian, that allows SimplEx to make explicit the contribution of each corpus feature in the mixture. Through experiments on tasks ranging from mortality prediction to image classification, we demonstrate that these decompositions are robust and accurate. With illustrative use cases in medicine, we show that SimplEx empowers the user by highlighting relevant patterns in the corpus that explain model representations. Moreover, we demonstrate how the freedom in choosing the corpus allows the user to have personalized explanations in terms of examples that are meaningful for them.
authors.name: Jonathan Crabbé
authors.name: Zhaozhi Qian
references.authors: Zachary C. Lipton
references.title: The Mythos of Model Interpretability
references.venue: Communications of the ACM,
references.authors: Travers Ching Daniel S. Himmelstein Brett K. Beaulieu-Jones Alexandr A. Kalinin Brian T. Do Gregory P. Way Enrico Ferrero Paul-Michael Agapow Michael Zietz Michael M. Hoffman Wei Xie Gail L. Rosen Benjamin J. Lengerich Johnny Israeli Jack Lanchantin Stephen Woloszynek Anne E. Carpenter Avanti Shrikumar Jinbo Xu Evan M. Cofer Christopher A. Lavender Srinivas C. Turaga Amr M. Alexandari Zhiyong Lu David J. Harris Dave DeCaprio Yanjun Qi Anshul Kundaje Yifan Peng Laura K. Wiley Marwin H.S. Segler Simina M. Boca S. Joshua Swamidass Austin Huang Anthony Gitter Casey S. Greene
references.title: Opportunities and obstacles for deep learning in biology and medicine
references.venue: Journal of The Royal Society Interface,
references.authors: Erico Tjoa Cuntai Guan
references.title: A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
references.venue: IEEE Transactions on Neural Networks and Learning Systems,
references.authors: Alejandro Barredo Arrieta Natalia Díaz-Rodríguez Javier Del Ser Adrien Bennetot Siham Tabik Alberto Barbado Salvador Garcia Sergio Gil-Lopez Daniel Molina Richard Benjamins Raja Chatila Francisco Herrera
references.title: Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI
references.venue: Information Fusion,
references.authors: Arun Das Paul Rad
references.title: Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
references.authors: Arun Rai
references.title: Explainable AI: from black box to glass box
references.venue: Journal of the Academy of Marketing Science
references.authors: Lloyd Shapley
references.title: A value for n-person games
references.venue: Contributions to the Theory of Games,
references.authors: Anupam Datta Shayak Sen Yair Zick
references.title: Algorithmic Transparency via Quantitative Input Influence: Theory and Experiments with Learning Systems
references.venue: In Proceedings - 2016 IEEE Symposium on Security and Privacy,
references.authors: Scott Lundberg Su-In Lee
references.title: A Unified Approach to Interpreting Model Predictions
references.venue: Advances in Neural Information Processing Systems,
references.authors: Marco Tulio Ribeiro Sameer Singh Carlos Guestrin
references.title: Why should i trust you?" Explaining the predictions of any classifier
references.venue: In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,
references.authors: Mukund Sundararajan Ankur Taly Qiqi Yan
references.title: Axiomatic Attribution for Deep Networks
references.venue: 34th International Conference on Machine Learning, ICML 2017,
references.authors: Amit Dhurandhar Pin-Yu Chen Ronny Luss Chun-Chen Tu Paishun Ting Karthikeyan Shanmugam Payel Das
references.title: Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negatives
references.venue: Advances in Neural Information Processing Systems,
references.authors: Ruth C. Fong Andrea Vedaldi
references.title: Interpretable Explanations of Black Boxes by Meaningful Perturbation
references.venue: Proceedings of the IEEE International Conference on Computer Vision,
references.authors: Ruth Fong Mandela Patrick Andrea Vedaldi
references.title: Understanding deep networks via extremal perturbations and smooth masks
references.venue: Proceedings of the IEEE International Conference on Computer Vision,
references.authors: Jonathan Crabbé Mihaela van der Schaar
references.title: Explaining Time Series Predictions with Dynamic Masks
references.venue: International Conference on Machine Learning,
references.authors: Giang Nguyen Daeyoung Kim Anh Nguyen
references.title: The effectiveness of feature attribution methods and its correlation with automatic evaluation scores
references.venue: Advances in Neural Information Processing Systems,
references.authors: R. Caruana H. Kangarloo J.D. Dionisio U. Sinha D. Johnson
references.title: Case-based explanation of non-case-based learning methods
references.venue: AMIA Symposium,
references.authors: Isabelle Bichindaritz Cindy Marling
references.title: Case-based reasoning in the health sciences: What’s next
references.venue: In Artificial Intelligence in Medicine,
references.authors: Mark T. Keane Eoin M. Kenny
references.title: How Case-Based Reasoning Explains Neural Networks: A Theoretical Analysis of XAI Using Post-Hoc Explanation-by-Example from a Survey of ANN-CBR Twin-Systems
references.venue: In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics),
references.authors: Been Kim Cynthia Rudin Julie Shah
references.title: The Bayesian Case Model: A Generative Approach for Case-Based Reasoning and Prototype Classification
references.venue: Advances in Neural Information Processing Systems,
references.authors: Been Kim Rajiv Khanna Oluwasanmi Koyejo
references.title: Examples are not Enough, Learn to Criticize! Criticism for Interpretability
references.venue: Advances in Neural Information Processing Systems,
references.authors: Karthik S. Gurumoorthy Amit Dhurandhar Guillermo Cecchi Charu Aggarwal
references.title: Efficient Data Representation by Selecting Prototypes with Importance Weights
references.venue: Proceedings - IEEE International Conference on Data Mining, ICDM,
references.authors: R. Dennis Cook Sanford Weisenberg
references.title: Residuals and influence in regression
references.authors: Pang Wei Koh Percy Liang
references.title: Understanding Black-box Predictions via Influence Functions
references.venue: 34th International Conference on Machine Learning, ICML 2017,
references.authors: Amirata Ghorbani James Zou
references.title: Data Shapley: Equitable Valuation of Data for Machine Learning
references.venue: 36th International Conference on Machine Learning,
references.authors: Amirata Ghorbani Michael P. Kim James Zou
references.title: A Distributional Framework for Data Valuation
references.authors: Been Kim Martin Wattenberg Justin Gilmer Carrie Cai James Wexler Fernanda Viegas Rory Sayres
references.title: Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)
references.venue: 35th International Conference on Machine Learning,
references.authors: Nicolas Papernot Patrick McDaniel
references.title: Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
references.authors: Chih-Kuan Yeh Joon Sik Kim Ian E.H. Yen Pradeep Ravikumar
references.title: Representer Point Selection for Explaining Deep Neural Networks
references.venue: Advances in Neural Information Processing Systems,
references.authors: Mark T. Keane Eoin M. Kenny
references.title: The Twin-System Approach as One Generic Solution for XAI: An Overview of ANN-CBR Twins for Explaining Deep Learning
references.authors: Li Deng
references.title: The mnist database of handwritten digit images for machine learning research
references.venue: IEEE Signal Processing Magazine,
references.authors: Eric Wu Kevin Wu Roxana Daneshjou David Ouyang Daniel E Ho James Zou
references.title: How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals
references.venue: Nature Medicine,
references.authors: Pan Ji Tong Zhang Hongdong Li Mathieu Salzmann Ian Reid
references.title: Deep Subspace Clustering Networks
references.venue: Advances in Neural Information Processing Systems,
references.authors: Rushil Anirudh Jayaraman J. Thiagarajan Rahul Sridhar Peer-Timo Bremer
references.title: MARGIN: Uncovering Deep Neural Networks using Graph
references.venue: Signal Analysis. arXiv,
references.authors: Gregory Cohen Saeed Afshar Jonathan Tapson André van Schaik
references.title: EMNIST: an extension of MNIST to handwritten letters
references.authors: Alberto Abadie
references.title: Using Synthetic Controls: Feasibility, Data Requirements, and Methodological Aspects
references.venue: Journal of Economic Literature,
references.authors: Alberto Abadie Alexis Diamond Jens Hainmueller
references.title: Synthetic control methods for comparative case studies: Estimating the effect of California’s Tobacco control program
references.venue: Journal of the American Statistical Association,
references.authors: Susan Athey Mohsen Bayati Nikolay Doudchenko Guido Imbens Khashayar Khosravi
references.title: Matrix Completion Methods for Causal Panel Data Models
references.authors: Muhammad Amjad Devavrat Shah Dennis Shen
references.title: Robust Synthetic Control
references.venue: Journal of Machine Learning Research,
references.authors: Jennifer Gordetsky Jonathan Epstein
references.title: Grading of prostatic adenocarcinoma: current state and prognostic implications
references.venue: Diagnostic Pathology,
sections.heading: 1 Introduction and related work
sections.text: How can we make a machine learning model convincing? If accuracy is undoubtedly necessary, it is rarely sufficient. As these models are used in critical areas such as medicine, finance and the criminal justice system, their black-box nature appears as a major issue [1, 2, 3]. With the necessity to address this problem, the landscape of explainable artificial intelligence (XAI) developed [4, 5]. A first approach in XAI is to focus on white-box models that are interpretable by design. However, restricting to a class of inherently interpretable models often comes at the cost of lower prediction accuracy [6]. In this work, we rather focus on post-hoc explainability techniques. These methods aim at improving the interpretability of black-box models by complementing their predictions with various kinds of explanations. In this way, it is possible to understand the prediction of a model without sacrificing its prediction accuracy.
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  Feature importance explanations are undoubtedly the most widespread type of post-hoc explanations. Popular feature importance methods include SHAP [7, 8, 9], LIME [10], Integrated Gradients [11], Contrastive Examples [12] and Masks [13, 14, 15]. These methods complement the model prediction for an input example with a score attributed to each input feature. This score reflects the importance of each feature for the model to issue its prediction. Knowing which features are important for a model prediction certainly provides more information on the model than the prediction by itself. However, these methods do not provide a reason as to why the model pays attention to these particular features.
  Another approach is to contextualize each model prediction with the help of relevant examples. In fact, recent works [16] have demonstrated that human subjects often find example-based explanations more insightful than feature importance explanations. Complementing the model’s predictions with relevant examples previously seen by the model is commonly known as Case-Based Reasoning (CBR) [17, 18, 19]. The implementations of CBR generally involve models that create a synthetic representation of the dataset, where examples with similar patterns are summarized by prototypes [20, 21, 22]. At inference time, these models relate new examples to one or several prototypes to issue a prediction. In this way, the patterns that are used by the model to issue a prediction are made explicit with the help of relevant prototypes. A limitation of this approach is the restricted model architecture. The aforementioned procedure requires to opt for a family of models that rely on prototypes to issue a prediction. This family of model might not always be the most suitable for the task at hand. This motivates the development of generic post-hoc methods that make few or no assumption on the model.
  The most common approach to provide example-based explanations for a wide variety of models mirrors feature importance methods. The idea is to complement the model prediction by attributing a score to each training example. This score reflects the importance of each training example for the model to issue its prediction. This score will typically be computed by simulating the effect of removing each training instance from the training set on the learned model [23]. Popular examples of such methods include Influence Functions [24] and Data-Shapley [25, 26]. These methods offer the advantage of being flexible enough to be used with a wide variety of models. They produce scores that describe what the model could have predicted if some examples were absent from the training set. This is very interesting in a data valuation perspective. However, in an explanation perspective, it is not clear how to reconstruct the model predictions with these importance scores.
  So far, we have only discussed works that provide explanations of a model output, which is the tip of the iceberg. Modern machine learning models involve many convoluted transformations to deduce the output from an input. These transformations are expressed in terms of intermediate variables that are often called latent variables. Some treatment of these latent variables is necessary if we want to provide explanations that take the model complexity into account. This motivates several works that push the explainability task beyond the realm of model outputs. Among the most noticeable contributions in this endeavour, we cite Concept Activation Vectors that create a dictionary between human friendly concepts (such as the presence of stripes in an image) and their representation in terms of latent vectors [27]. Another interesting contribution is the Deep k-Nearest Neighbors model that contextualizes the prediction for an example with its Nearest Neighbours in the space of latent variables, the latent space [28]. An alternative exploration of the latent space is offered by the representer theorem that allows, under restrictive assumptions, to use latent vectors to decompose a model’s prediction in terms of its training examples [29].
  Contribution In this work, we introduce a novel approach called SimplEx that lies at the crossroad of the above research directions. SimplEx outputs post-hoc explanations in the form of Figure 1, where the model’s prediction and latent representation for a test example is approximated as a mixture of examples extracted from a corpus of examples.
  In each case, SimplEx highlights the role played by each feature of each corpus example in the latent space decomposition. SimplEx centralizes many functionalities that, to the best of our knowledge, constitute a leap forward from the previous state of the art. (1) SimplEx gives the user freedom to choose the corpus of examples whom with the model’s predictions are decomposed. Unlike previous methods such as the representer theorem, there is no need for this corpus of examples to be equal to the model’s training set. This is particularly interesting for two reasons: (a) the training set of a model is not always accessible (b) the user might want explanations in terms of examples that make sense for them. For instance, a doctor might want to understand the predictions of a risk model in terms of patients they know. (2) The decompositions of SimplEx are valid, both in latent and output space. We show that, in both cases, the corpus mixtures discovered by SimplEx offer significantly more precision and robustness than previous methods such as Deep k-Nearest Neighbors and the representer theorem. (3) SimplEx details the role played by each feature in the corpus mixture. This is done by introducing Integrated Jacobians, a generalization of Integrated Gradients that makes the contribution of each corpus feature explicit in the latent space decomposition. This creates a bridge between two research directions that have mostly developed independently: feature importance and example-based explanations [19, 30]. In Section 3 of the supplementary material, we report a user-study involving 10 clinicians. This study supports the significance of our contribution.
sections.heading: 2 SimplEx
sections.text: In this section, we formulate our method rigorously. Our purpose is to explain the black-box prediction for an unseen test example with the help of a set of known examples that we call the corpus. We start with a clear statement of the family of black-boxes for which our method applies. Then, we detail how the set of corpus examples can be used to decompose a black-box representation for the unseen example. Finally, we show that the corpus decomposition can offer explanations at the feature level.
sections.heading: 2.1 Preliminaries
sections.text: Let X ⊆ RdX be an input (or feature) space and Y ⊆ RdY be an output (or label) space, where dX and dY are respectively the dimension of the input and the output space. Our task is to explain individual predictions of a given black-box f : X → Y . In order to build our explainability method, we need to make an assumption on the family of black-boxes that we wish to interpret. Assumption 2.1 (Black-box Restriction). We restrict to black-boxes f : X → Y that can be decomposed as f = l ◦ g, where g : X → H maps an input x ∈ X to a latent vector h = g (x) ∈ H and l : H → Y linearly maps1 a latent vector h ∈ H to an output y = l(h) = Ah ∈ Y . In the following, we call H ⊆ RdH the latent space. Typically, this space has higher dimension than the output space dH > dY . Remark 2.1. In the context of deep-learning, this assumption requires that the last hidden layer maps linearly to the output. While it is often the case, it is crucial in the following since we will use the fact that linear combinations in latent space correspond to linear combinations in output space. Our purpose is to gain insights on the structure of the latent space. Remark 2.2. This assumption is compatible with regression and classification models, we just need to clarify what we mean by output in the case of classification. If f is a classification black-box that predicts the probabilities for each class, it will typically take the form in Assumption 2.1 up to a normalizing map φ (typically a softmax): f = φ ◦ l ◦ g. In this case, we ignore2 the normalizing map φ and define the output to be y = (l ◦ g) (x). Our explanations for f rely on a set of examples that we call the corpus. These examples will typically (but not necessarily) be a representative subset of the black-box training set. The corpus set has to be understood as a set of reference examples that we want to use as building blocks to interpret unseen examples. In order to index these examples, it will be useful to denote by [n1 : n2] the set of natural numbers between the natural numbers n1 and n2 with n1 < n2. Further, we denote [n] = [1 : n] the set of natural numbers between 1 and n ≥ 1. The corpus of examples is a set C = {xc | c ∈ [C]} containing C ∈ N∗ examples xc ∈ X . In the following, superscripts are labels for examples and subscripts are labels for vector components. In this way, xci has to be understood as the component i of corpus example c.
  1The map can in fact be affine. In the following, we omit the bias term b ∈ Y that can be reabsorbed in g. 2There is no loss of information as the output allows us to reconstruct class probabilities p via p = φ (y).
sections.heading: 2.2 A corpus of examples to explain a latent representation
sections.text: Our purpose is to understand a prediction f(x) for an unseen test example x with the help of the corpus. How can we decompose the prediction f(x) in terms of corpus predictions f(xc)? A naive attempt would be to express x as a mixture of inputs from the corpus C: x = ∑Cc=1 wcxc with weights wc ∈ [0, 1] that sum to one ∑Cc=1 wc = 1. The weakness of this approach is that the signification of the mixture weights is not conserved if the black-box f is not a linear map: f( ∑C c=1 w cxc) 6=∑Cc=1 wcf(xc).
  Fortunately, Assumption 2.1 offers us a better vector space to perform a corpus decomposition of the unseen example x. We first note that the map g induces a latent representation of the corpus g(C) = {hc = g(xc) | xc ∈ C} ⊂ H. Similarly, x has a latent representation h = g(x) ∈ H. Following the above line of reasoning, we could therefore perform a corpus decomposition in latent space h = ∑C c=1 w
  chc. Now, by using the linearity of l, we can compute the black-box output of this mixture in latent space: l( ∑C c=1 w chc) = ∑C c=1 w cl(hc). In this case, the weights that are used to decompose the latent representation h in terms of the latent representation of the corpus g(C) also reflect the way in which the black-box prediction f(x) can be decomposed in terms of the corpus outputs f(C). This hints that the latent spaceH is endowed with the appropriate geometry to make corpus decompositions. More formally, we think in terms of the convex hull spanned by the corpus. Definition 2.1 (Corpus Hull). The corpus convex hull spanned by a corpus C with latent representation g(C) = {hc = g(xc) | xc ∈ C} ⊂ H is the convex set
  CH (C) = {
  C∑ c=1 wchc ∣∣∣∣∣ wc ∈ [0, 1] ∀c ∈ [C] ∧ C∑ c=1 wc = 1 } .
  Remark 2.3. This is the set of latent vectors that are a mixture of the corpus latent vectors.
  At this stage, it is important to notice that an exact corpus decomposition is not possible if h 6∈ CH(C). In such a case, the best we can do is to find the element ĥ ∈ CH(C) that best approximates h. IfH is endowed with a norm ‖ · ‖H, this corresponds to the convex optimization problem
  ĥ = argmin h̃ ∈ CH(C) ‖ h− h̃ ‖H . (1)
  By definition, the corpus representation ĥ of h can be expanded3 as a mixture of elements from g(C): ĥ =∑Cc=1 wchc. The weight can naturally be interpreted as a measure of importance in the reconstruction of h with the corpus. Clearly, wc ≈ 0 for some c ∈ [C] indicates that hc does not play a significant role in the corpus representation ĥ of h. On the other hand, wc ≈ 1 indicates that hc generates the corpus representation ĥ by itself.
  At this stage, a natural question arises: how can we know if the corpus approximation ĥ is a good approximation for h? The answer is given by the residual vector h− ĥ that measures the shift between the latent representation h = g(x) and the corpus hull CH(C). It is natural to use this residual vector to detect examples that cannot be explained with the selected corpus of examples C. Definition 2.2 (Corpus Residual). The corpus residual associated to a latent vector h ∈ H and its corpus representation ĥ ∈ CH(C) solving (1) is the quantity
  rC(h) =‖ h− ĥ ‖H= min h̃ ∈ CH(C) ‖ h− h̃ ‖H .
  gx1
  x2 x3
  h1
  h2
  h3x h
  ĥ
  Figure 2: Corpus convex hull and residual.
  In Section 1.1 of the supplementary material, we show that the corpus residual also controls the quality of the corpus approximation in output space Y . All the corpus-related quantities that we have introduced so far are summarized visually in Figure 2. Note that this Figure is a simplification of the reality as C will typically
  be larger than 3 and dX , dH will typically be higher than 2. We are now endowed with a rigorous way to decompose a test example in terms of corpus examples in latent space. In the next section, we detail how to pull-back this decomposition to input space.
  3Note that this decomposition might not be unique, more details in Section 1 of the supplementary material.
sections.heading: 2.3 Transferring the corpus explanation in input space
sections.text: Now that we are endowed with a corpus decomposition ĥ = ∑C
  c=1 w chc that approximates h, it
  would be convenient to have an understanding of the corpus decomposition in input space X . For the sake of notation, we will assume that the corpus approximation is good so that it is unnecessary to draw a distinction between the latent representation h of the unseen example x and its corpus decomposition ĥ. If we want to understand the corpus decomposition in input space, a natural approach [11] is to fix a baseline input x0 together with its latent representation h0 = g(x0). Let us now decompose the representation shift h− h0 in terms of the corpus:
  h− h0 = C∑
  c=1
  wc ( hc − h0 ) . (2)
  With this decomposition, we understand the total shift in latent space h− h0 in terms of individual contributions from each corpus member. In the following, we focus on the comparison between the baseline and a single corpus example xc together with its latent representation hc by keeping in mind that the full decomposition (2) can be reconstructed with the whole corpus. To bring the discussion in input spaceX , we interpret the shift in latent space hc−h0 as resulting from a shift xc−x0 in the input space. We are interested in the contribution of each feature to the latent space shift. To decompose the shift in latent space in terms of the features, we parametrize the shift in input space with a line γc : [0, 1] → X that goes from the baseline to the corpus example: γc(t) = x0 + t · (xc − x0) for t ∈ [0, 1]. Together with the black-box, this line induces a curve in latent space g ◦ γc : [0, 1]→ H that goes from the baseline latent representation h0 to the corpus example latent representation hc. Let us now use an infinitesimal decomposition of this curve to make the contribution of each input feature explicit. If we assume that g is differentiable at γc(t), we can use a first order approximation of the curve at the vicinity of t ∈ (0, 1) to decompose the infinitesimal shift in latent space:
  g ◦ γc(t+ δt)− g ◦ γc(t)︸ ︷︷ ︸ Infinitesimal shift in latent space = dX∑ i=1 ∂g ∂xi ∣∣∣∣ γc(t) dγci dt ∣∣∣∣ t δt+ o(δt)
  = dX∑ i=1 ∂g ∂xi ∣∣∣∣ γc(t) (xci − x0i ) · δt+ o(δt),
  where we used γci (t) = x 0 i + t · (xci − x0i ) to obtain the second equality. In this decomposition, each input feature contributes additively to the infinitesimal shift in latent space. It follows trivially that the contribution of the input feature corresponding to input dimension i ∈ [dX ] is given by
  δjci (t) = (x c i − x0i ) · ∂g ∂xi ∣∣∣∣ γc(t) δt ∈ H.
  In order to compute the overall contribution of feature i to the shift, we let δt→ 0 and we sum the infinitesimal contributions along the line γc. If we assume4 that g is almost everywhere differentiable, this sum converges to an integral in the limit δt→ 0 . This motivates the following definitions. Definition 2.3 (Integrated Jacobian & Projection). The integrated Jacobian between a baseline (x0,h0 = g(x0)) and a corpus example (xc,hc = g(xc)) ∈ X ×H associated to feature i ∈ [dX ] is
  jci = ( xci − x0i ) ∫ 1 0 ∂g ∂xi ∣∣∣∣ γc(t) dt ∈ H,
  where γc(t) ≡ x0 + t · ( xc − x0 ) for t ∈ [0, 1]. This vector indicates the shift in latent space induced by feature i of corpus example c when comparing the corpus example with the baseline. To summarize this contribution to the shift h− h0 described in (2), we define the projected Jacobian
  pci = projh−h0 (j c i ) ≡ 〈 h− h0 , jci 〉 〈 h− h0 , h− h0 〉
  ∈ R,
  where 〈·,·〉 is an inner product forH and the normalization is chosen for the purpose of Proposition 2.1. Remark 2.4. The integrated Jacobian can be seen as a latent-space generalization of Integrated Gradients [11]. In Section 1.3 of the supplementary material, we establish the relationship between the two quantities: IGci = l(j c i ).
  4This is not restrictive, DNNs with ReLU activation functions satisfy this assumption for instance.
  We summarize the Jacobian quantities in Figure 3. By inspecting the figure, we notice that projected Jacobians encode the contribution of feature i from corpus example c to the overall shift in latent space: pci > 0 implies that this feature creates a shift pointing in the same direction as the overall shift; pci < 0 implies that this feature creates a shift pointing in the opposite direction and pci = 0 implies that this feature creates a shift in an orthogonal direction. We
  use the projections to summarize the contribution of each feature in Figures 1 , 8 & 9. The colors green and red indicate respectively a positive and negative projection. In addition to these geometrical insights, Jacobian quantities come with natural properties.
  Proposition 2.1 (Properties of Integrated Jacobians). Consider a baseline (x0,h0 = g(x0)) and a test example together with their latent representation (x,h = g(x)) ∈ X × H. If the shift h − h0 admits a decomposition (2), the following properties hold.
  (A) : C∑
  c=1 dX∑ i=1 wcjci = h− h0 (B) : C∑ c=1 dX∑ i=1 wcpci = 1.
  Proof. The proof is provided in Section 1.4 of the supplementary material.
  These properties show that the integrated Jacobians and their projections are the quantities that we are looking for: they transfer the corpus explanation into input space. The first equality decomposes the shift in latent space in terms of contributions wcjci arising from each feature of each corpus example. The second equality sets a natural scale to the contribution of each feature. For this reason, it is natural to use wcpci to measure the contribution of feature i of corpus example c.
sections.heading: 3 Experiments
sections.text: In this section, we evaluate quantitatively several aspects of our method. In a first experiment, we verify that the corpus decomposition scheme described in Section 2 yields good approximations for the latent representation of test examples extracted from the same dataset as the corpus examples. In a realistic clinical use case, we illustrate the usage of SimplEx in a set-up where different corpora reflecting different datasets are used. The experiments are summarized below. In Section 2 of the supplementary material, we provide more details and further experiments with time series and synthetic data. The code for our method and experiments is available on the Github repository https://github.com/JonathanCrabbe/Simplex. All the experiments have been replicated on different machines.
sections.heading: 3.1 Precision of corpus decomposition
sections.text: Description The purpose of this experiment is to check if the corpus decompositions described in Section 2 allows us to build good approximations of the latent representation of test examples. We start with a datasetD that we split into a training setDtrain and a testing setDtest. We train a black-box f for a given task on the training set Dtrain. We randomly sample a set of corpus examples from the training set C ⊂ Dtrain (we omit the true labels for the corpus examples) and a set of test examples from the testing set T ⊂ Dtest. For each test example x ∈ T , we build an approximation ĥ for h = g(x) with the corpus examples latent representations. In each case, we let the method use only K corpus examples to build the approximation. We repeat the experiment for several values of K.
  Metrics We are interested in measuring the precision of the corpus approximation in latent space and in output space. To that aim, we use the R2 score in both spaces. In this way, R2H measures the precision of the corpus approximation ĥ with respect to the true latent representation h. Similarly, R2Y measures the precision of the corpus approximation ŷ = l(ĥ) with respect to the true output y = l(h). Both of these metrics satisfy −∞ < R2 ≤ 1. A higher R2 score is better with R2 = 1
  corresponding to a perfect approximation. All the metrics are computed over the test examples T . The experiments are repeated 10 times to report standard deviations across different runs.
  Baselines We compare our method5 (SimplEx) with 3 baselines. A first approach, inspired by [28], consists in using the K-nearest corpus neighbours in latent space to build the latent approximation ĥ. Building on this idea, we introduce two baselines (1) KNN Uniform that takes the average latent representation of the K-nearest corpus neighbours of h in latent space (2) KNN Distance that computes the same average with weights wc inversely proportional to the distance ‖ h − hc ‖H. Finally, we use the representer theorem [29] to produce an approximation ŷ of y with the corpus C. Unlike the other methods, the representer theorem does not allow to produce an approximation in latent space.
  Datasets We use two different datasets with distinct tasks for our experiment: (1) 240,486 patients enrolled in the American SEER program [31]. We consider the binary classification task of predicting cancer mortality for patients with prostate cancer. We train a multilayer perceptron (MLP) for this task. Since this task is simple, we show that a corpus of C = 100 patients yields good approximations. (2) 70,000 MNIST images of handwritten digits [32]. We consider the multiclass classification task of identifying the digit represented on each image. We train a convolutional neural network (CNN) for the image classification. This classification task is more complex than the previous one (higher dX and dY ), we show that a corpus of C = 1, 000 images yields good approximations in this case.
  Results The results for SimplEx and the KNN baselines are presented in Figure 4 & 5. Several things can be deduced from these results: (1) It is generally harder to produce a good approximation in latent space than in output space as R2H < R 2 Y for most examples (2) SimplEx produces the most accurate approximations, both in latent and output space. These approximations are of high quality with R2 ≈ 1. (3) The trends are qualitatively different between SimplEx and the other baselines. The accuracy of SimplEx increases with K and stabilizes when a small number of corpus members contribute (K = 5 in both cases). The accuracy of the KNN baselines increases with K,
  5To enforce SimplEx to select K examples, we add a L1 penalty making the C −K smallest weights vanish.
  reaches a maximum for a small K and steadily decreases for larger K. This can be understood easily: whenK increases beyond the number of relevant corpus examples, irrelevant examples will be added in the decomposition. SimplEx will typically annihilate the effect of these irrelevant examples by setting their weights wc to zero in the corpus decomposition. The KNN baselines include the irrelevant corpus members in the decomposition, which alters the quality of the approximation. This suggests that K has to be tuned for each example with KNN baselines, while the optimal number of corpus examples to contribute is learned by SimplEx. (4) The standard deviations indicate that the performances of SimplEx are more consistent across different runs. This is particularly true in the prostate cancer experiment, where the corpus size C is smaller. This suggests that SimplEx is more robust than the baselines. (5) For the representer theorem, we have R2Y = −(6.6± 6.1) · 107 for the prostate cancer dataset and R2Y = −(7.2± 6.6) for MNIST. This corresponds to poor estimations of the black-box output. We propose some hypotheses to explain this observation in Section 2.1 of the supplementary material.
sections.heading: 3.2 Significance of Jacobian Projections
sections.text: Description The purpose of this experiment is to check if SimplEx’s Jacobian Projections are a good measure of the importance for each corpus feature in constructing the test latent representation h. In the same setting as in the previous experiment, we start with a corpus C of C = 500 MNIST images. We build a corpus approximation for an example x ∈ X with latent representation h = g(x) ∈ H. The precision of this approximation is reflected by its corpus residual rC(h). For each corpus example xc ∈ C, we would like to identify the features that are the most important in constructing the corpus decomposition of h. With SimplEx, this is reflected by the Jacobian Projections pci . We evaluate these scores for each feature i ∈ [dX ] of each corpus example c ∈ [C]. For each corpus image xc ∈ C, we select the n most important pixels according to the Jacobian Projections and the baseline. In each case, we build a mask mc that replaces these n most important pixels by black pixels. This yields a corrupted corpus image xccor = mc xc, where denotes the Hadamard product. By corrupting all the corpus images, we obtain a corrupted corpus Ccor . We analyse how well this corrupted corpus approximates h, this yields a residual rCcor (h).
  Metric We are interested in measuring the effectiveness of the corpus corruption. This is reflected by the metric δcor(h) = rCcor (h)− rC(h) . A higher value for this metric indicates that the features selected by the saliency method are more important for the corpus to produce a good approximation of h in latent space. We repeat this experiment for 100 test examples and for different numbers n of perturbed pixels.
  Baseline As a baseline for our experiment, we use Integrated Gradients, which is close in spirit to our method. In a similar fashion, we compute the Integrated Gradients IGci for each feature i ∈ [dX ] of each corpus example c ∈ [C] and construct a corrupted corpus based on these scores. Results The results are presented in the form of box plots in Figure 6. We observe that the corruptions induced by the Jacobian Projections are significantly more impactful when few pixels are perturbed. The two methods become equivalent when more pixels are perturbed. This demonstrates that Jacobian Projections are more suitable to measure the importance of features when performing a latent space reconstruction, as it is the case for SimplEx.
sections.heading: 3.3 Use case: clinical risk model across countries
sections.text: Very often, clinical risk models are produced and validated with the data of patients treated at a single site [33]. This can cause problems when these models are deployed at different sites for two reasons: (1) Patients from different sites can have different characteristics (2) Rules that are learned for one site might not be true for another site. One possible way to alleviate this problem would be to detect patients for which the model prediction is highly extrapolated and/or ambiguous. In this way, doctors from different sites can make an enlightened use of the risk model rather than blindly believing the model’s predictions. We demonstrate that SimplEx provides a natural framework for this set-up.
  As in the previous experiment, we consider a dataset DUSA containing patients enrolled in the American SEER program [31]. We train and validate an MLP risk model with DUSA. To give a realistic realization of the above use-case, we assume that we want to deploy this risk model in a different site: the United Kingdom. For this purpose, we extract DUK from the set of 10,086 patients enrolled in the British Prostate Cancer UK program [34]. These patients are characterized by the
  same features for both DUK and DUSA. However, the datasets DUK and DUSA differ by a covariate shift: patients from DUK are in general older and at earlier clinical stages. When comparing the two populations in terms of the model, a first interesting question to ask is whether the covariate shift between DUSA and DUK affects the model representation. To explore this question, we take a first corpus of American patients CUSA ⊂ DUSA. If there is indeed a difference in terms of the latent representations, we expect the representations of test examples fromDUK to be less closely approximated by their decomposition with respect to CUSA. If this is true, the corpus residuals associated to examples of DUK will typically larger than the ones associated to DUSA. To evaluate this quantitatively, we consider a mixed set of test examples T sampled from both DUK and DUSA: T ⊂ DUK t DUSA. We sample 100 examples from both sources: | T ∩ DUK| = |T ∩ DUSA| = 100. We then approximate the latent representation of each example h ∈ g(T ) and compute the associated corpus residual rCUSA(h). We sort the test examples from T by decreasing order of corpus residual and we use this sorted list to see if we can detect the examples from DUK. We use previous baselines for comparison, results are shown in Figure 7.
  Several things can be deduced from this experiment. (1) The results strongly suggest that the difference between the two datasets DUSA and DUK is reflected in their latent representations. (2) The corpus residuals from SimplEx offer the most reliable way to detect examples that are different from the corpus examples CUSA. None of the methods matches the maximal baseline since some examples of DUSA resemble examples from DUK. (3) When the corpus examples are representative of the training set, as it is the case in the experiment, our approach based on SimplEx provides a systematic way to detect test examples that have representations that are different from the ones produced at training time. A doctor should be more sceptical with respect to model predictions associated
  to larger residual with respect to CUSA as these arise from an extrapolation region of the latent space. Let us now make the case more concrete. Suppose that an American and a British doctor use the above risk model to predict the outcome for their patients. Each doctor wants to decompose the predictions of the model in terms of patients they know. Hence, the American doctor selects a corpus of American patients CUSA ⊂ DUSA and the British doctor selects a corpus of British patients CUK ⊂ DUK. Both corpora have the same size CUSA = CUK = 1, 000. We suppose that the doctors know the model prediction and the true outcome for each patient in their corpus. Both doctors are sceptical about the risk model and want to use SimplEx to decide when it can be trusted. This leads them to a natural question: is it possible to anticipate misclassification with the help of SimplEx?
  In Figure 8 & 9, we provide two typical examples of misclassified British patients from DUK \ CUK together with their decomposition in terms of the two corpora CUSA and CUK. These two examples
  exhibit two qualitatively different situations. In Figure 8, both the American and the British doctors make the same observation: the model relates the test patient to corpus patients that are mostly misclassified by the model. With the help of SimplEx, both doctors will rightfully be sceptical with respect to the model’s prediction.
  In Figure 9, something even more interesting occurs: the two corpus decompositions suggest different conclusions. In the American doctor’s perspective, the prediction for this patient appears perfectly coherent as all patients in the corpus decomposition have very similar features and all of them are rightfully classified. On the other hand, the British doctor will reach the opposite conclusion as the most relevant corpus patient is misclassified by the model. In this case, we have a perfect illustration of the limitation of the transfer of a risk model from one site (America) to another (United Kingdom): similar patients from different sites can have different outcomes. In both cases, since the test patient is British, only the decomposition in terms of CUK really matters. In both cases, the British doctor could have anticipated the misclassification of each patient with SimplEx.
sections.heading: 4 Discussion
sections.text: We have introduced SimplEx, a method that decomposes the model representations at inference time in terms of a corpus. Through several experiments, we have demonstrated that these decompositions are accurate and can easily be personalized to the user. Finally, by introducing Integrated Jacobians, we have brought these explanations to the feature level.
  We believe that our bridge between feature and example-based explainability opens up many avenues for the future. A first interesting extension would be to investigate how SimplEx can be used to understand latent representations involved in unsupervised learning. For instance, SimplEx could be used to study the interpretability of self-expressive latent representations learned by autoencoders [35]. A second interesting possibility would be to design a rigorous scheme to select the optimal corpus for a given model and dataset. Finally, a formulation where we allow the corpus to vary on the basis of observations would be particularly interesting for online learning.
sections.heading: Acknowledgement
sections.text: The authors are grateful to Alicia Curth, Krzysztof Kacprzyk, Boris van Breugel, Yao Zhang and the 4 anonymous NeurIPS 2021 reviewers for their useful comments on an earlier version of the manuscript. Jonathan Crabbé would like to thank Bogdan Cebere for replicating the experiments in this manuscript. Jonathan Crabbé would like to acknowledge Bilyana Tomova for many insightful discussions and her constant support. Jonathan Crabbé is funded by Aviva. Fergus Imrie is supported by by the National Science Foundation (NSF), grant number 1722516. Zhaozhi Qian and Mihaela van der Schaar are supported by the Office of Naval Research (ONR), NSF 1722516.
title: Explaining Latent Representations with a Corpus of Examples