summaries: The premise of the paper is that example train instances with the corresponding label can be an effective explanation of the model's prediction rather than post-hoc explanation which is not deterministic. The framework jointly trains label prediction and prototype clustering. Then, the framework computes the similarity between input and prototypes to retrieve the most similar prototype as an explanation while predicting the label as well. Moreover, the framework can reflect human-in-the-loop feedback on the prototype. The primary result of the paper is that their framework can show the proper explanation by a similar example while preserving the performance. This paper aims to model explanation and task prediction such that task performances are not (or less) traded off for interpretability. It proposes a novel framework for transformer models where classification and explanation generation are based on shared prototype embeddings which are learnt from training data by a combination of losses. The framework is also compatible with settings that requires human in the loop for extra supervision on prototype learning. Experiment results show that adding the proposed ProtoTrex benefit task performances on 3 sentiment classification tasks. This paper proposes Proto-Trex model to increase the interpretability of the text classification systems. The proposed model mainly adds a bunch of prototype layers to learn the similarity between the query and prototypes. They also propose an extension caleld iProto-Trex to interactively learn from users' feedbacks. Experimental results and example cases show that the Proto-Trex could give comparable classification accuracy compared to plain classification model (w/o explanation) and could provide reasonable explanations.  This paper introduces a method for improving interpretability of black box transformer based text classifiers. The approach is based on “case based reasoning” where the network classifies an input by comparing it against a library of learnt prototypes (each with a corresponding label) and classifying the input based on a weighted similarity with the prototypes. Thus, in a sense it is more interpretable than an end-to-end classifier since a user can directly look at the similarity scores with the prototypes to understand the labeling decision. The approach also promises to be more “faithful” than post-hoc interpretability methods since the predictions are based directly on similarity scores. The paper also shows that end-users of the system can interact with it by either editing prototypes (if they are experts) or providing weak feedback about which prototypes are good, and this feedback can be integrated to re-learn better prototypes. From experiments we see that their approach is competitive with standard end-to-end finetuning while being more interpretable. Based on sufficiency and comprehensiveness scores, we see some evidence that the explanations from prototypes are faithful, though it is unclear if these scores are competitive. 
abstractText: Transformer language models are state-of-the-art in a multitude of NLP tasks. Despite these successes, their opaqueness remains problematic. Recent methods aiming to provide interpretability and explainability to black-box models primarily focus on post hoc explanations of (sometimes spurious) input-output correlations. Instead, we emphasize using prototype networks directly incorporated into the model architecture and hence explain the reasoning process behind the network’s decisions. Moreover, while our architecture performs on par with several language models, it enables one to learn from user interactions. This not only offers a better understanding of language models but uses human capabilities to incorporate knowledge outside of the rigid range of purely data-driven approaches.
references.authors: Pepa Atanasova Jakob Grue Simonsen Christina Lioma Isabelle Augenstein
references.title: A diagnostic study of explainability techniques for text classification
references.venue: In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),
references.authors: Yonatan Belinkov James Glass
references.title: Analysis methods in neural language processing: A survey
references.venue: Transactions of the Association for Computational Linguistics (TACL),
references.authors: Emily M. Bender Timnit Gebru Angelina McMillan-Major Shmargaret Shmitchell
references.title: On the dangers of stochastic parrots: Can language models be too big
references.venue: Conference on Fairness, Accountability, and Transparency (FAccT),
references.authors: Chaofan Chen Oscar Li Daniel Tao Alina Barnett Cynthia Rudin Jonathan K Su
references.title: This looks like that: Deep learning for interpretable image recognition
references.venue: In Proceedings of the 2019 Conference on Advances in Neural Information Processing Systems (NeurIPS),
references.authors: James R. Clifton Geoffrey Frohnsdorff
references.title: Applications of computers and information technology - 18. In Handbook of Analytical Techniques in Concrete Science and Technology, pp. 765–799
references.venue: William Andrew Publishing,
references.authors: Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova. Bert
references.title: Pre-training of deep bidirectional transformers for language understanding
references.venue: In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ((NAACL)),
references.authors: Jay DeYoung Sarthak Jain Nazneen Fatema Rajani Eric Lehman Caiming Xiong Richard Socher Byron C. Wallace
references.title: ERASER: A benchmark to evaluate rationalized NLP models
references.venue: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,
references.authors: Shuoyang Ding Philipp Koehn
references.title: Evaluating saliency methods for neural language models
references.venue: In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL),
references.authors: Kawin Ethayarajh
references.title: How contextual are contextualized word representations? comparing the geometry of bert, elmo, and GPT-2 embeddings
references.venue: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),
references.authors: Mor Geva Roei Schuster Jonathan Berant Omer Levy
references.title: Transformer feed-forward layers are key-value memories
references.venue: arXiv preprint arXiv:2012.14913,
references.authors: Mor Geva Uri Katz Aviv Ben-Arie Jonathan Berant
references.title: What’s in your head? emergent behaviour in multi-task transformer models
references.venue: arXiv preprint arXiv:2104.06129,
references.authors: Jonathan Gordon Benjamin Van Durme
references.title: Reporting bias and knowledge acquisition
references.venue: In Proceedings of the 2013 Workshop on Automated Knowledge Base Construction (AKBC),
references.authors: Xiaochuang Han Byron C. Wallace Yulia Tsvetkov
references.title: Explaining black box predictions and unveiling data artifacts through influence functions
references.venue: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL),
references.authors: Peter Hase Chaofan Chen Oscar Li Cynthia Rudin
references.title: Interpretable image recognition with hierarchical prototypes
references.venue: Proceedings of the AAAI Conference on Human Computation and Crowdsourcing,
references.authors: Alon Jacovi Yoav Goldberg
references.title: Aligning Faithful Interpretations with their Social Attribution
references.venue: Transactions of the Association for Computational Linguistics,
references.authors: Sarthak Jain Byron C. Wallace
references.title: Attention is not Explanation
references.venue: In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL),
references.authors: Pang Wei Koh Percy Liang
references.title: Understanding black-box predictions via influence functions
references.venue: In Proceedings of the 34th International Conference on Machine Learning (ICML),
references.authors: Oscar Li Hao Liu Chaofan Chen Cynthia Rudin
references.title: Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions
references.venue: In Proceedings of the 2018 Conference on Artificial Intelligence (AAAI),
references.authors: Christopher D. Manning Prabhakar Raghavan Hinrich Schütze
references.title: Introduction to Information Retrieval
references.authors: Yao Ming Panpan Xu Huamin Qu Liu Ren
references.title: Interpretable and steerable sequence learning via prototypes
references.venue: Proceedings of the 25th International Conference on Knowledge Discovery and Data Mining (KDD),
references.authors: Bo Pang Lillian Lee Shivakumar Vaithyanathan
references.title: Thumbs up? sentiment classification using machine learning techniques
references.venue: In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP),
references.authors: Pouya Pezeshkpour Sarthak Jain Byron C. Wallace Sameer Singh
references.title: An empirical comparison of instance attribution methods for NLP
references.venue: In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL),
references.authors: Alec Radford Jeff Wu Rewon Child David Luan Dario Amodei Ilya Sutskever
references.title: Language models are unsupervised multitask learners
references.authors: Alec Radford Jong Wook Kim Chris Hallacy Aditya Ramesh Gabriel Goh Sandhini Agarwal Girish Sastry Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever
references.title: Learning transferable visual models from natural language supervision
references.venue: Proceedings of the 38th International Conference on Machine Learning,ICML
references.authors: Nils Reimers Iryna Gurevych
references.title: Sentence-bert: Sentence embeddings using siamese bertnetworks
references.venue: In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP),
references.authors: Marco Tulio Ribeiro Sameer Singh Carlos Guestrin
references.title: why should i trust you?”: Explaining the predictions of any classifier
references.venue: In Proceedings of the 22nd International Conference on Knowledge Discovery and Data Mining (KDD),
references.authors: Andrew Slavin Ross Michael C. Hughes Finale Doshi-Velez
references.title: Right for the right reasons: Training differentiable models by constraining their explanations
references.venue: In Proceedings of the TwentySixth International Joint Conference on Artificial Intelligence (IJCAI),
references.authors: Victor Sanh Lysandre Debut Julien Chaumond Thomas Wolf
references.title: Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter
references.venue: arXiv preprint arXiv:1910.01108,
references.authors: Patrick Schramowski Wolfgang Stammer Stefano Teso Anna Brugger Xiaoting Shao Hans-Georg Luigs Anne-Katrin Mahlein Kristian Kersting
references.title: Making deep neural networks right for the right scientific reasons by interacting with their explanations
references.venue: Nat Mach Intell,
references.authors: Ramprasaath Ramasamy Selvaraju Stefan Lee Yilin Shen Hongxia Jin Shalini Ghosh Larry P. Heck Dhruv Batra Devi Parikh
references.title: Taking a HINT: leveraging explanations to make vision and language models more grounded
references.venue: In 2019 International Conference on Computer Vision (ICCV),
references.authors: Daniel Smilkov Nikhil Thorat Been Kim Fernanda Viégas Martin Wattenberg
references.title: Smoothgrad: removing noise by adding noise
references.venue: arXiv preprint arXiv:2012.09699,
references.authors: Wolfgang Stammer Patrick Schramowski Kristian Kersting
references.title: Right for the right concept: Revising neuro-symbolic concepts by interacting with their explanations
references.venue: In Proceedings of the 2021 Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Mukund Sundararajan Ankur Taly Qiqi Yan
references.title: Axiomatic attribution for deep networks
references.venue: In Proceedings of the 34th International Conference on Machine Learning (ICML),
references.authors: Ian Tenney James Wexler Jasmijn Bastings Tolga Bolukbasi Andy Coenen Sebastian Gehrmann Ellen Jiang Mahima Pushkarna Carey Radebaugh Emily Reif Ann Yuan
references.title: The language interpretability tool: Extensible, interactive visualizations and analysis for NLP models
references.venue: In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP): System Demonstrations,
references.authors: Stefano Teso Kristian Kersting
references.title: Explanatory interactive machine learning
references.venue: In Proceedings of the 2019 Conference on AI, Ethics, and Society (AIES),
references.authors: Elena Voita Rico Sennrich Ivan Titov
references.title: The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives
references.venue: In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),
references.authors: Elena Voita David Talbot Fedor Moiseev Rico Sennrich Ivan Titov
references.title: Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned
references.venue: In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL),
references.authors: Sarah Wiegreffe Yuval Pinter
references.title: Attention is not not explanation
references.venue: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),
sections.heading: 1 INTRODUCTION
sections.text: Transformer language models (LMs) are ubiquitous in NLP today but also notoriously opaque. Therefore, it is not surprising that a growing body of work aims to interpret them: Recent evaluations of approaches of saliency methods (Ding & Koehn, 2021) and instance attribution methods Pezeshkpour et al. (2021) find that, while intriguing, for the same outputs, different methods assign importance to different inputs. Furthermore, they are usually employed post hoc, thus possibly encouraging reporting bias (Gordon & Van Durme, 2013). The black-box character of LMs becomes especially problematic, as the data to train them might be unfiltered and contain (human) bias. As a result, ethical concerns about these models arise, which can have a substantial negative impact on society as they get increasingly integrated into our lives (Bender et al., 2021).
  Here, we focus on providing case-based reasoning explanations during the inference process, directly outputting the LM’s predictions. In doing so, we avoid the problems mentioned above of post hoc explanations and help to reduce the issue of (human) bias. To increase the interpretability of the model, we enhance the transformer architecture with a prototype layer and propose PrototypicalTransformer Explanation (Proto-Trex) Networks. Proto-Trex networks provide an explanation as a prototypical example for a specific model prediction, which is similar to (training-)samples with the corresponding label.
  Our experimental results demonstrate that Proto-Trex networks perform on par with noninterpretable baselines, e.g. BERT (Devlin et al., 2019) and GPT (Radford et al., 2019). In terms of explanations, we show promising results with learned prototypes providing helpful explanations for the user to understand better the LMs decision-making, which, in turn, increases trust and reliability. To further enhance the prototypical network with human supervision, we propose an interactive learning setting (iProto-Trex) that allows users of any knowledge to give feedback and improve the model (Ribeiro et al., 2016), moving beyond a purely data-driven approach.
  To summarize, our contributions are as follows: We (i) introduce prototype networks for transformer LMs that provide explanations and (ii) show that they are on par with non-interpretable baselines on classification tasks on different architectures. Furthermore, to improve prototype networks, we (iii) provide a novel interactive prototype learning setup accounting for user feedback certainty.
  We proceed as follows. We start by briefly reviewing related work of interpretability in NLP. Then we introduce Proto-Trex networks, including our novel interactive learning setup combining explanatory interactive learning with prototype networks. Before concluding, we discuss faithful explanations and touch upon the results of our experimental evaluation.
sections.heading: 2 TOWARDS THE EXPLAINABILITY OF TRANSFORMER LANGUAGE MODELS
sections.text: To open the black-box of transformers, we use i/Proto-Trex networks built upon post hoc interpretation methods, case-based reasoning approaches, and explanatory interactive learning.
  Post hoc interpretability. Various (post hoc) interpretability methods focus on different parts of the transformer architecture. Atanasova et al. (2020) and Belinkov & Glass (2019) provide overviews of this fast developing field. Generally, there are methods that analyze word representations (Voita et al., 2019a), the attention distribution throughout the model (Jain & Wallace, 2019; Wiegreffe & Pinter, 2019), and the (attention and classification) model heads (Voita et al., 2019b; Geva et al., 2021). Other approaches such as Geva et al. (2020) focus on the feed-forward layers. Gradientbased approaches, such as Sundararajan et al. (2017) and Smilkov et al. (2017) can generally be used to trace gradients, while influence functions (Koh & Liang, 2017) trace model parameter changes throughout a LM (Han et al., 2020). While backtracking all the model weights might be possible, such explanations can only tell us which part of the input they are looking at (Chen et al., 2019) and humans are ill-equipped to interpret them (Stammer et al., 2021). So instead, we aim for more intuitive and sparse explanations: well-descriptive but short sequences as prototypes. These are not only able to reveal the parts of the input it is looking at but also show prototypical cases similar to those parts.
  Case-based Reasoning in Deep Neural Networks. Our work relates most closely to previous work demonstrating the benefits of prototype networks in Computer Vision (Li et al., 2018; Chen et al., 2019) as well as for sequential data by combining them with RNNs (Hase et al., 2019; Ming et al., 2019). More precisely, in contrast to post hoc interpretation methods, prototype networks employ explanations in the, e.g. classification process, i.e. classify a sample by comparing several parts – which could be words of a sentence– to (learned) prototypical parts from other samples for a given class. If things look similar, they are classified similarly. Humans behave very similarly, which is called case-based reasoning (Clifton & Frohnsdorff, 2001). The prototypical parts provided by the network help the user to understand the classification. Tab. 1 illustrates the benefit of prototypical explanations. One can observe that the post hoc explanation can leave the user clueless, while the prototypical explanations can help better understand the network’s decision. Accordingly, prototypes are an additional method in the interpretability toolbox, extending current post hoc methods. As described, the reasoning process of prototypical networks is qualitatively similar to that of humans (Clifton & Frohnsdorff, 2001; Chen et al., 2019). In previous approaches, however, the training of such networks was entirely data-driven. Therefore, like Chen et al. (2019), in our work, we do not focus on quantifying unit interpretability of prototypical networks, but extend learning the reasoning process of our network by Explanatory Interactive Learning (Schramowski et al., 2020).
  Explanatory Interactive Learning (XIL) incorporates explanatory supervision in the learning process by involving the human user. More precisely, human users can ask the model to explain a prediction. They can then respond by correcting the model if necessary, providing a slightly improved –but not necessarily optimal– feedback on the explanations (Stammer et al., 2021). Interactive learning and, in particular, explanatory interactive learning showed not only to increase the explanation quality of deep black-box models but also the trust of the user (Ross et al., 2017; Selvaraju et al.,
  2019; Teso & Kersting, 2019; Schramowski et al., 2020). In contrast to previous (post hoc) XIL methods, we avoid tracing gradients. That makes our approach particularly efficient, as we only change prototypes (cf. Fig 4).
sections.heading: 3 PROTO-TREX: PROTOTYPE LEARNING FOR TRANSFORMER LMS
sections.text: The prototype network (Proto-Trex) architecture builds upon large-scale transformer LMs, summarized in Fig. 2. In the following, we describe the general idea of prototypical networks and our transformer-based prototype architecture and its modules in more detail.
sections.heading: 3.1 PROTO-TREX NETWORKS
sections.text: General Idea. Proto-Trex is a prototype network, where prototypes are representative observations in the training set. The classification is based on the neighboring prototypes, i.e. an input query is classified as positive when the network thinks it looks similar to a certain (near) positive prototype within the training set. In more detail, the reasoning and classification processes use the same module, i.e. , they use the similarity between query and prototypes. As both reasoning and classification use the same module (in parallel), we consequently have an interpretable model. In particular, (interpretable) prototypical explanations help verify Right for the Right Reasons (Ross et al., 2017).
  Architecture. Specifically, the model, in our case the pre-trained transformer f , creates a context embedding e from the input sequence x. This embedding contains useful features for the prediction and is then passed on to the prototype layer. This layer computes the similarity between the embedding e and each of the trainable prototypes pj from the set of prototypes P . The prototypes are learned during the training process and represent prototypical (sub-)sequences in the training data. We use transformer models both on the sentence-level (a) and the word-level (b), resulting in a single input representation for the sentence-level transformer and l for word-level. To compare the word-level embeddings ei with the prototypes, we have to patch them (Z = {patch(ei)}) into subsequences according to the desired prototype length k. The resulting similarity values are passed on to the final linear layer g with weight matrix wg and no bias, mapping from m (the number of prototypes) to the number of classes. Finally, the output values are passed on to a softmax producing class probabilities and predictions t.
sections.heading: 3.2 PROTO-TREX LOSS
sections.text: Optimization of prototype networks aims at maximizing both performance and interpretability. To this end, our Proto-Trex loss L combines performance and interpretability with prior knowledge of the prototype network’s structure. Let us illustrate this for the word-level case (the sentence-level case results from e=z): L :=
  min P,wg
  1
  n n∑ i=1 CE(ti, yi) + λ1Clst(z,p) + λ2Sep(z,p) + λ3Distr(z,p) + λ4Divers(p) + λ5||wg||
  (1)
  where λi weights the influence of the different terms and n is the number of training examples. The first term is the cross-entropy (CE) loss optimizing the predictive power of the network. The second term (Clst) clusters the prototypes w.r.t. the training examples of the same class, maximizing similarity to them (we rewrite maximization as minimization terms), and the separation loss minimizes the similarity to other-class instances:
  Clst(z,p) := − 1 n n∑ i=1 min j:pj∈Pyi min z∈Z sim(z,pj) ; Sep(z,p) := 1 n n∑ i=1 min j:pj /∈Pyi min z∈Z sim(z,pj) (2)
  Together, Clst and Sep push each prototype to focus more on training examples from the same class and less on training examples from other classes. Both are motivated by ProtoPNet (Chen et al., 2019). To get prototypes that are distributed well in the embedding space, we introduce two additional losses, a distribution loss (Li et al., 2018), assuring that a prototype is nearby each training example, and a diversity loss (Ming et al., 2019):
  Distr(z,p) := − 1 m m∑ j=1 min i∈[1,n] min z∈Z sim(z,pj) ; Divers(p) := 1 m m∑ ȷ̂=1 min j:pj∈P sim(pȷ̂,pj) (3)
  In contrast to the other terms, the diversity loss does not compute similarities between embeddings and prototypes but between prototypes themselves. It is another way of distributing prototypes in the embedding space as it maximizes the distance between prototypes, preventing them from staying at the same –not necessarily optimal– location. This is especially helpful in the case of multiple prototypes for a single class, encouraging them to represent different facets of the class. If they are otherwise too similar, no information is gained and resulting in redundant prototypes – together with the class-specific loss encouraged by the cluster loss (Clst) and the separation loss (Sep), this helps to compute prototypes that focus solely on their class. Otherwise, we can get ambiguous prototypes leading to negative reasoning. Also, we clamp the weights of the classification layer with min(wg, 0) to avoid negative reasoning (Chen et al., 2019). Finally, the last term of the ProtoTrex loss (Eq. 1) is an L1-regularization term of the last layer (g), which prevents the network from overfitting or relying too much on a single prototype.
sections.heading: 3.3 SIMILARITY COMPUTATION
sections.text: Computing similarities is an essential aspect of Proto-Trex. For prototypes to represent certain aspects or features of the input distribution in the embedding space, we compute the similarity sim(e,p) between an embedded training example and a prototype. A distance minimization can replace each similarity maximization max sim(e,p) = min dist(e,p). We here follow common practice and explore both the L2-norm or the cosine similarity:
  sim(e,pj) =
  { −∥e− pj∥2 , L2-norm
  e·pj ∥e∥2∥pj∥2
  , cosine similarity ,
  where the index j denotes a specific prototype. For each training example, there is an embedding e which is compared to all m prototypes p, i.e. we get m similarity values for each embedding.
  The L2-norm assumes a Gaussian prior, which can be a wrong assumption, which is why we also investigate cosine similarity. While L2-norm computes the distance between two vectors, cosine similarity measures the angle between them. Both, but especially cosine similarity, are natural choices for NLP tasks (Manning et al., 2008). However, cosine similarity is more robust here than L2-norm as the distance’s magnitude between vectors has no influence due to normalization.
sections.heading: 3.4 SELECTION FOR WORD-LEVEL PROTOTYPES
sections.text: Since learning prototypes for LMs pre-trained on word-level representations is more involved than for the sentence-level, let us focus on them here; the sentence-level case naturally follows from the discussion. Word-level prototypes are generally sensible as explanations should consist of sparse sequences, at best focusing only on subsequences of the input sentence. As sequences can also be ambiguous and contain little information, we combine different word embeddings (patch()) and enforce the prototype to be similar to the relevant subsequences, containing the most information. But how do we select the most informative words?
  A naive approach would be to simply compute all possible patches (word combinations) of the input sequence and compare them to the prototypes to find the best patch for classification – the most important subsequence should then be similar to a certain prototype. Unfortunately, for long input sequences with length l it becomes hard to compute all possible word patches of length k, in this case |Z|= ( l k ) . To solve this problem without losing too many valuable patches, the following two approaches are sensible ideas.
  (a) Sliding windows naturally reduce the number of patches. A sliding window is a convolution that selects a certain part of the input according to the window size and then sliding to the next part. The main disadvantage of sliding windows is the relatively rigid structure of a window. This is problematic in the context of NLP tasks that often have long-range dependencies. We introduce dilation to loosen this. Dilation facilitates looking at direct word neighborhoods but also at more distant ones to capture long-range dependencies. This “convolutional” approach is illustrated in Fig. 3(a). We note that applied to the contextualized word embeddings of transformer LMs, the convolutional approach without dilation should contain not only local information but also some global information. Unfortunately, this information is stored in the embeddings and cannot be visualized easily.
  As an alternative, the (b) Self-Attention approach adds a self-attention layer after the transformer f and before the prototype layer p to filter irrelevant words (with low attention-scores), cf. Fig. 3(b). The purpose of this self-attention layer differs from the one in the transformer LM itself. The here used attention mechanism selects the most important words, and the embedding representation remains untouched. To still provide as much information and variety as possible, the number of selected words nw of the attention layer which are passed into the distance computation is a hyperparameter, and we chose it to be twice as much as the length of a single prototype however clamped by the threshold klim: nw =min(max(klim, k), 2k). The threshold can be set w.r.t. computational efficiency. This means having a prototype of length k=4 and klim =10, the attention layer selects nw=8 words yielding in this example |Z|= ( 8 4 ) =70 patches for the distance computation.
sections.heading: 3.5 DECODING VIA NEAREST NEIGHBOR PROJECTION
sections.text: Proto-Trex networks encode prototypes in the embedding space. Consequently, they cannot simply be decoded from the transformer embedding space, as this space and textual data are categorical and not continuous. To overcome this, we assign, i.e. project, each prototype to its nearest neighbor in the training data (where the textual representation is available) in an intermediate training step. Thereby, we ensure that the prototype really represents what it actually looks like, which also increases the interpretability of the Proto-Trex network. The final training step (after the nearest neighbor projection) then fine-tunes the classification head to adapt it to the projected prototypes. Doing so has the advantage of being decoded precisely on the spot and providing more certainty for the explanatory power. The disadvantage, however, is that a prototype may be located sub-optimally, which could lead to performance losses. We faced this issue especially for word-level prototypes and found the projection ineligible for these highly contextualized word representations. Instead, for word-level, we only use the nearest neighbor for approximating the prototype to provide the explanation.
sections.heading: 4 INTERACTIVE PROTOTYPE LEARNING
sections.text: Recent case-based reasoning approaches in Computer Vision and NLP usually assume a large volume of data for training (Li et al., 2018; Chen et al., 2019; Hase et al., 2019), with little or no
  user feedback during the model building process. However, user knowledge and interaction, in particular via explanations, can be valuable already in the model building and understanding phase, significantly reducing the amount of data required, avoiding Clever-Hans moments early on, and increasing the explanation quality of the model and, in turn, the trust of the user, see e.g. Teso & Kersting (2019); Schramowski et al. (2020).
  Interactive Proto-Trex. To address this, we propose an explanatory interactive learning approach for Proto-Trex networks, called iProto-Trex, as illustrated in Fig. 4. Specifically, carried out during the training progress by freezing and evaluating the current state or post hoc training, the interaction takes the following form. At each step, the Proto-Trex networks provide prototypes as explanations of a classification. The user responds by correcting the learner, if necessary, in our case, the prototypes. For this, iProto-Trex provides a range of opportunities for interacting with prototypes.
  Specifically, iProto-Trex distinguishes between weak-knowledge and strong-knowledge interactions. In strong-knowledge interactions, users are certain about their feedback (which could require high-level expertise). In this case, iProto-Trex provides options to remove and add prototypes. Removing a prototype is also helpful if the network has learned redundant prototypical sequences or prototypes that cover all critical aspects of the task at hand. If users are content with all the present prototypical sequences, they can also add new prototypes (Replacing a prototype is the same as removing plus adding a prototype). After these interactions, the subsequent classification layer is retrained to integrate and balance the new prototype setup in its decision-making.
  In weak-knowledge interactions, users only state content or discontent with a prototype based on their intuition. They do not need to know what a replacement should exactly look like, which is a good trade-off between user knowledge and loss optimization of the network. To this end, iProto-Trex uses re-initialization and fine-tuning; instead of providing an explicit replacement, users freeze prototypes they like while relearning those they are dissatisfied with. The difference between these approaches is that users can express how well the current prototype represents a particular task aspect. Another form of weak-knowledge interaction, which iProto-Trex provides only for sentencelevel prototypes, is pruning the sequence length of prototypical explanations, i.e. , compressing it to the essentials. In order to limit meaning changes of a sequence, a threshold is provided for the cosine similarity of the pruned and the original version. We initially set this threshold to 0.8, but users can set the threshold as they like, resulting in a strong-knowledge interaction.
  Soft User Feedback. iProto-Trex’s interaction methods, especially the strong-knowledge ones, require users to be quite confident about their feedback –they have to be experts. To elevate this burden, we propose a soft feedback mechanism, using a loss based on the prototype similarity as
  Linteract := λ6 max pold
  ( max(sim(pold,pnew), c) ) (4)
  with c ∈ [0, 1]. Instead of directly replacing the selected prototype pold, this soft interaction loss pushes the prototypes to be similar to the desired prototype pnew as suggested by a user. Moreover, users can control how strong iProto-Trex should incorporate their interventions by setting a certainty value c between 0 (low certainty) and 1 (high certainty).
sections.heading: 5 FAITHFULNESS OF PROTOTYPICAL EXPLANATIONS
sections.text: A question that naturally arises when dealing with explainable AI is how to evaluate the quality of an explanation. One of the key metrics in NLP and Computer Vision is the faithfulness of explanations
  (Jacovi & Goldberg, 2021; DeYoung et al., 2020), i.e. whether a given explanation represents the true reasoning process of the black-box model. We apply and adjust this metric to text-based prototype networks. First, we evaluate the faithfulness of Proto-Trex as a whole (DeYoung et al., 2020) by showing that the class probabilities change significantly if we perturb the input. We follow their approach of computing comprehensiveness and sufficiency. To calculate comprehensiveness, we remove the rationale in each input sequence and evaluate the changes in the class probabilities for the prediction. Second, we perturb the prototype layer by removing the top explanation for a given test sample and compute the loss in accuracy. This helps to identify to what extent the decision was actually based on the explanation given. Since our prototypical explanations are already short sequences, removing the rationale in the explanation is similar to removing the entire prototype.
sections.heading: 6 EXPERIMENTAL EVALUATION
sections.text: Our intention here is to investigate how good prototypes help to understand transformer LMs. To this end, we evaluated i/Proto-Trex explanations on three benchmark datasets: MovieReview (Pang et al., 2002), Open Yelp1 and Jigsaw Toxicity2. We compared five pre-trained LMs (GPT-2 (Radford et al., 2019), BERT (Devlin et al., 2019), DistilBERT, (Sanh et al., 2020), SBERT (Reimers & Gurevych, 2019) and the text-encoder of CLIP (Radford et al., 2021)) to investigate three questions:
  (Q1) How much does adding a prototype layer affect the performance of (non-interpretable) LMs, i.e. , a classification head defined by two fully-connected non-linear layers?
  (Q2) How does the performance change after and during interaction with the model explanations? In particular, we investigated the different modules of Proto-Trex networks on sentenceand word-level and the interaction between users and the prototypical explanations.
  (Q3) How faithful are the given explanations, i.e. how well does the given explanation represent the true reasoning of the black-box model?
  We present qualitative and quantitative results and refer to the Appendix for additional details on the experiments and our implementation, as well as additional qualitative results. If not stated otherwise, the Proto-Trex architecture includes sentence- and word-level embeddings, the convolution module without dilation to select word-tokens, and cosine similarity to compute the similarity between prototypical explanations and input query. We optimized the prototype and classification module with the proposed loss (Eq. 1) and initialize the prototypes randomly. In each experiment, the number of prototypes is m=10, and the largest variant of the corresponding pre-trained LM is evaluated.
  (Q1,2) Trade off accuracy and interpretability. Tab. 1 summarizes the experimental result of Proto-Trex network based on different sentence- and word-level LMs. As one can see and expected from the literature, interpretability comes along with a trade-off in accuracy. The trade-off is generally higher on sentence-level LMs, partially due to the nearest neighbor projection (cf. Appendix for direct comparison). However, the difference between traditional LMs and Proto-Trex LMs is often marginal and task-dependent (e.g. DistilBERT on Yelp and Movie). Surprisingly, in the case of BERT and GPT-2 (on Yelp and Toxicity), the Proto-Trex network is outperforming the baseline LMs. Most interestingly, one can see that the user may boost the performance of the corresponding Proto-Trex network interactively (iProto-Trex). Overall, i/Proto-Trex is competitive with state-ofthe-art LMs while being much more transparent.
  (Q1) Ablation study. Next, we investigate different Proto-Trex module choices for performance, cf. Tab. 1b), and explanation provision impact. While input- and prototype similarity computation does not explicitly correlate with the prototypes learned –yet we assumed it to be the better choice–, the word-selection module also impacts the explanation outcome. Furthermore, cosine similarity is not only more accurate but also converges faster. In terms of accuracy, the convolutional word selection outperforms the attention module. More importantly, we also observe an advantage for the provided explanations; namely, the attention module tends to select punctuation and stop words. According to Ethayarajh (2019), this is because punctuation and stop-words are among the most context-specific word representations–they are not polysemous themselves but can have an infinite number of contexts. In contrast, the convolution module is easier to interpret as words are coherent.
  1https://www.yelp.com/dataset 2https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge
  (Q1,2) Provided explanations. To analyze the provided explanations, we consider the SBERT based Proto-Trex networks, cf. Tab. 1, trained on the Open Yelp dataset. Tab. 2 shows how ProtoTrex provides users with explanations. These explanations are given as prototypical sequences that correspond to a query. In addition, Proto-Trex provides corresponding importance scores, indicating the significance of an explanation for the classification. We show four sentence-level prototypes for the query that users can quickly extract the important aspects that help understand the classification. However, one can observe that the sentence-level explanations are sometimes lacking sparsity and are difficult to interpret about the query, which also demonstrates the demand for (further) interaction (cf. colored boxes with prototypical explanations P6 and P2 in Tab. 2). The pruned Proto-Trex provides less ambiguous and easier to interpret sequences.
  (Q2) Interactive prototype learning. In the previous experiment, pruning has already shown the benefits of adapting explanations to users’ preferences. In order to further evaluate our interactive learning setup, we incorporated certainty (c) and evaluated its effectiveness. Since sentence-level
  prototypes allow for better coherence, this is where we will focus the interactive learning. Tab. 3 shows an influential, i.e. high importance value, sentence-level explanation (yet no interaction) for negative restaurant reviews on the Open Yelp dataset. Assuming a user is dissatisfied with an explanation yet uncertain about what a good explanation would entail: Applying re-initialization (most uncertain interaction technique) confirms the user’s intuition of a “weak” component, as the slight increase in accuracy indicates. More importantly, we can already observe that users can influence the network’s decision process based on their preferences without performance loss. However, the revised explanation is still not sufficiently interpretable. Therefore, we considered incorporating explicit user feedback here. In this case, the phrase “They offer a bad service” served as soft replacement for the model’s provided explanation, and the user applied it with different levels of certainty. First, a low certainty value (c=0.5) results in the same explanation as before. This is because the similarity of any two prototypes with clearly negative sentiment is higher than a certainty threshold of 0.5. As the user gets more certain, he gradually increases the threshold. Finally (c = 1), the user simply replaced the prototype to obtain his desired solution without a trade-off in accuracy. In summary, our results demonstrate that interactive learning is a solid method to counter the network’s incapability in consistently providing interpretable prototypes along with high accuracy.
  (Q3) Faithful prototypes. To analyze faithfulness, we first follow DeYoung et al. (2020) and use the MovieReview dataset as they provide human-annotated rationales for this dataset. We compare the classification probabilities of the samples (DeYoung et al., 2020) with the ones where the rationale is removed (comprehensiveness) and with the ones with only the rationale (sufficiency). Tab. 1(c) shows that i/Proto-Trex scores high for comprehensiveness, indicating that the network is focusing on the input rationale for the classification, while low sufficiency scores indicate that the surrounding context has little to even poor impact on the prediction. This shows that i/Proto-Trex are generally faithful classifiers, i.e. the human-annotated rationales agree with the internal rationales in the model. Furthermore, we remove the top (prototype) explanation for the classification and examine the change in accuracy after the removal to show that it is the prototype that faithfully captures the rationale. Tab. 1(c) shows a drastic decrease in accuracy for both models after the explanation removal, confirming the faithful contribution of prototypical explanations to the classification.
sections.heading: 7 CONCLUSION
sections.text: Large-scale transformer LMs, like other black-box models, lack interpretability. We presented methods (prototype networks) to incorporate case-based reasoning to explain the LM’s decisions. Despite the explanatory power of prototypical explanations, challenges regarding the quality of their interpretability still exist. Previous applications lack human supervision, although case-based reasoning is a human-inspired approach. Therefore, we propose an interactive prototype learning setup to overcome these challenges and improve the network’s capabilities by incorporating human knowledge with the consideration of knowledge certainty. Future work covers improving methods for patching the input for word-level prototypes to enable projection for higher interpretability.
sections.heading: 8 ETHICS AND REPRODUCIBILITY STATEMENT
sections.text: The article presented here approaches the explainability of transformer language models through case-based reasoning in the form of prototypical explanations. The interpretable prototype module that we learned is built on a pre-trained transformer language model. As these models are trained on data that is not publicly available, it remains an open question whether any kind of bias is inherent to the model as a whole (Bender et al., 2021). In addition, concerns about privacy violations and other potential misuse emerge as these models are trained with weak to no supervision.
  The code to reproduce the results can be found in our publicly available repository. Furthermore, we provide the trained Proto-Trex models to obtain explanations.
sections.heading: A APPENDIX
sections.text: A.1 DATASETS
  The benchmark datasets mentioned above provide a fixed test set except the Yelp Open dataset. For the Yelp Open dataset we randomly select 200 000 training examples and split them into train (70%), validation (15%) and test (15%) set. We split the Jigsaw Toxicity train set into train (20%) and validation (80%) set. Before the split, we filter out long sequences (num tokens > 40) for each dataset. This is required because transformer-based LMs can only handle sequences of limited
  length, especially CLIP, and long sequences also cause the other sequences of a set to be padded to the same length, which, in turn, produces many padding tokens. For the tokenization we use the GPT2-tokenizer3. We apply grid search for hyperparameters optimization. We report the crossvalidated results. For the faithfulness computation we use the eraserbenchmark repository4 and data5.
  A.2 TRAINING
  The Proto-Trex networks are trained with and evaluated on different datasets. We use PyTorch6 for the implementation. We optimize our model with Adam optimizer with hyperparameters β = (0.9, 0.999) and ϵ = 10−8. We use a base learning rate of lrbase = 0.001 and apply learning rate warm-up and scheduling that is here a linear decay. The learning rate is then given as
  lr = lrbase ·min ( stepi ewup , e− stepi e− ewup ) ,
  where stepi is the current step, e the total number of epochs and ewup the number of warm-up epochs. The warm-up takes place for ewup=min ( 10, e20 ) epochs. Warm-up reduces the dependency of early optimization steps that may cause difficulties in the longer run. Also, we weigh the cross-entropy loss with the class occurrences to re-balance the influence of an unbalanced dataset. The same is done in the accuracy computation where a balanced class accuracy is computed. For regularization, we apply an L1-regularization on the weights of the last linear layer. During the training process, a network may tend to overfit. To counteract this, we evaluate the model every 10th epoch on the validation set and keep the model that yielded the best validation result. Furthermore, we set up a class mask to assign each class to the same number of prototypes. This is used and enforced by the class-specific losses (Clst and Sep). Changing the balance of the class assignment can be sensible to correct for imbalance in the dataset or focus on a specific class if its sentiment is more polysemous or generally more relevant. We show the hyperparameters used for our experiments, found by a grid search. The found hyperparameters were often very similar across models and datasets as the parameters mainly aimed at interpretability, not only performance. Note, not all hyperparameters were grid-searched simultaneously. We first searched for the best combination of λ1 to λ5 and searched for λ6 separately because the interaction loss depends on interaction and does not directly impact the model performance during training. The best performing value for λ6 was 0.5. For the attention-based word selection we choose the number of heads to be 1 and k=4; klim=10.
  A.3 SENTENCE- VS. WORD-LEVEL LOSS
  For clarification, we present the loss terms for sentence-level i/Proto-Trex networks in more detail: L :=
  min P,wg
  1
  n n∑ i=1 CE(ti, yi) + λ1Clst(e,p) + λ2Sep(e,p) + λ3Distr(e,p) + λ4Divers(p) + λ5||wg|| ,
  (5)
  where
  Clst(e,p) := − 1 n n∑ i=1 min j:pj∈Pyi sim(ei,pj) , (6)
  Sep(e,p) := 1
  n n∑ i=1 min j:pj /∈Pyi sim(ei,pj) , (7)
  Distr(e,p) := − 1 n n∑ j=1 min i∈[1,n] sim(ei,pj) , (8)
  3https://huggingface.co/transformers/model_doc/gpt2.html#gpt2tokenizer 4https://github.com/jayded/eraserbenchmark 5https://www.eraserbenchmark.com/ 6https://pytorch.org/
  and
  Divers(p) := 1
  m m∑ ȷ̂=1 min j:pj /∈Pyi sim(pȷ̂,pj) . (9)
  A.4 RESULTS WITH CONFIDENCE INTERVALS
  Due to a lack of space we shifted the results with confidence intervals to the Appendix which you can see in tab. 5.
  A.5 PROJECTION
  We show the impact of projecting the prototypes onto their nearest neighbor in Tab. A.5. The projection is evaluated for sentence-level Proto-Trex networks. One can see the trade-off between interpretability and accuracy introduced by projection.
  A.6 PROVIDED EXPLANATIONS
  Here we show the explanations that Proto-Trex provides. For each network, we use 10 prototypes, and for word-level networks, we use a prototype length of 4 tokens. First we extend the results of Tab. 2 for the word-level case, shown in Tab. 7. Then we show prototype lists for the models on different datasets. In Tab. 8 we show all prototypes for Proto-Trex based on the GPT-2 transformer and in Tab. 9 the prototypes based on the SBERT transformer. Both tables show the prototypes for the Yelp Open and MovieReview dataset. The results for the Jigsaw Toxicity dataset can be found in an external document in the codebase. " CONTENT WARNING: the content in this document can be disturbing due to highly toxic texts! These results do not represent the authors’ opinion and show prototypical explanations provided solely by Proto-Trex.
  Tab. 10 shows the pruned prototypes of iProto-Trex from Tab. 9(a). Pruning cuts off the words at the end of a sequence that go beyond two sentences or 15 tokens in total.
  We showcase all experiments of the user interaction in Tab. 11 which is an extension of Tab. 3. The extension includes (1) retraining the classification layer for the same number of epochs to provide a fairer comparison with the interaction methods and to exclude changes in accuracy simply due to training for more epochs, (2) pruning the original, (3) removing the original, (4) adding a new prototype without changing the original, (5) replacing the original with the user-chosen alternative “They offer a bad service.” without using the interaction loss and (6) fine-tuning the original prototype.
  In Fig. 12 we shows the full explanations with importance scores for the exemplary query in the motivation (cf. Fig. 1). It highlights the advantage of using prototype networks and the benefit of interactive learning.