Abstract In order to better understand the reason behind model behaviors (i.e., making predic-tions), most recent work has exploited generative models to provide complementary explanations. However, existing approaches in natural language processing (NLP) mainly focus on WHY A rather than contrastive WHY A NOT B, which is shown to be able to better distinguish confusing candidates and improve model performance in other research fields. In this paper, we focus on generating C ontrastive E xplanations with counterfactual examples in NLI and propose a novel K nowledgeA ware generation framework ( KACE ). Specifically, we first identify rationales (i.e., key phrases) from input sentences, and use them as key perturbations for generating counterfactual examples. After obtaining qualified counterfactual examples, we take them along with original examples and external knowledge as input, and employ a knowledge-aware generative pre-trained language model to generate contrastive explanations. Experimental results show that contrastive explanations are bene-ficial to clarify the difference between predicted answer and other answer options. Moreover, we train an BERT-large based NLI model enhanced with contrastive explanations and achieve an accuracy of 91.9% on SNLI, gaining an improvement of 5.7% against ETPA (Explain-Then-Predict-Attention) and 0.6% against NILE (WHY A). In recent years, pre-trained language models (De-vlin et al., 2019; Liu et al., 2019; Yang et al., 2019) have been widely adopted in many tasks of natural language processing (Talmor et al., 2019; Choi et al., 2018; Bowman et al., 2015). However, due to  Work is done during internship at Alibaba Group. the lack of textual explanations, most downstream models become more complicated and difficult to understand. End users, especially those working in critical domains such as healthcare or online ed-ucation, become more skeptical and reluctant to adopt or trust them, although these models have been proved to improve the decision-making performance. Therefore, providing faithful textual explanations has become a promising way to overcome the black-box property of neural networks, which has attracted the attention of academia and industrial communities. Recently, the majority of existing methods (Xu et al., 2020; Cheng et al., 2020; Karimi et al., 2020; Ramamurthy et al., 2020; Atanasova et al., 2020; Kumar and Talukdar, 2020) in natural language processing try to explain the predictions of neural models in a model-intrinsic or model-agnostic (also known as post-hoc) way. While post-hoc models (Chen et al., 2020b; Karimi et al., 2020; Kumar and Talukdar, 2020) provide explanations after making predictions without affecting the overall accuracy, most of them neglect the rationales in inputs and provide textual explanations just in the form of WHY A. However, we argue that contrastive explanations in the form of WHY A NOT B could provide more informative and important clues that are easier to understand and persuade end-users. Moreover, we believe that contrastive explanations could benefit downstream tasks (e.g., NLI), since such kind of explanations contain more helpful information (e.g. relations between rationales) that can be used to improve model performance. To further enhance the explainability and performance of NLI, we propose a novel textual contrastive explanation generation framework in this paper, which is post-hoc and considers rationales, counterfactual examples, and external knowledge. Specifically, we first identify rationales (i.e., key phrases) from a premise-hypothesis (P-H) pair with Rationale Identification CounterfactualExampleGeneration CounterfactualExampleSelection ContrastiveExplanationGeneration Premise : A woman and a young child are making sculptures out of clay. Then we further select one most qualified counterfactual example for any other label B. Note that the acquisition of a qualified counterfactual example of class B is essential to generate a meaningful explanation for WHY NOT B, otherwise the resultant contrastive explanation will be groundless or useless. After that, we take the selected examples along with the original P-H pair and related external knowledge as input, and finally employ a knowledge-aware pre-trained language model to generate contrastive explanation, which will specify why the prediction label is A rather than B, and clarify the confusions for end-users. Moreover, we train an NLI model enhanced with contrastive explanations and achieve the new state-of-art performance on SNLI. The contributions of this paper are as follows:  We introduce a novel knowledge-aware contrastive explanation generation framework ( KACE ) for natural language inference tasks. We consider the rationales in inputs and regard them as important perturbations for generating counterfactual examples rather than just discarding them like previous post-hoc work (Hendricks et al., 2018; Cheng et al., 2020). We integrate external knowledge with generative pre-trained language model rather than only taking original inputs (Kumar and Talukdar, 2020; Rajani et al., 2019) for contrastive explanation generation. Experimental results show that knowledge-aware contrastive explanations are able to clarify the difference between predicted class and the others, which help to clarify the confusion of end-users and further improve model performance than WHY A explanations 1 . Here, we define the task of contrastive explanation generation for NLI. Given a trained neural network model f with input x and predicted class A , the problem of generating contrastive explanations (CE) to an input x is to specify why x belongs to category/class A rather than B , defined as: r = Rationales ( x, A ) (1) x (cid:48) = Reversal ( x, B, r ) (2) CE = Generator ( x (cid:48) , x, A ) (3) In Equation 1, we first identify a set of rationales in given inputs, as described in Section 3.1, and in Equation 2 we generate counterfactual examples with reversal mechanism as presented in Section 3.2. In Equation 3, we take the selected counterfactual example along with original example and external knowledge as input, and employ a knowledge-aware generator to produce contrastive explanation as detailed in Section 3.3. 1 Our code will be released as soon as possible at https://github.com/AI4NLP/KACE Long, fascinating, soulful. Considering that rationales are important features of an instance, it is essential to regard rationales as key perturbations for counterfactual example generation. In this paper, we formulate rationale identification as a token-level sequence labelling task where 1 indicates a rationale token and 0 indicates a background token. Being similar with (Thorne et al., 2019), we first construct the input sequence for a premise p and a hypothesis h as S p = (cid:104) s (cid:105) Label (cid:104) s (cid:105) P remise (cid:104) s (cid:105) and S h = (cid:104) s (cid:105) Hypothesis (cid:104) s (cid:105) , where (cid:104) s (cid:105) is a special token that separates the components. Let y represent the relation between S p and S h where y  { entailment, contradiction, neutral } . For each instance, we need to identify a subset r of zero or more tokens as rationales from both premise and hypothesis sentences. Both premise and hypothesis are encoded with RoBERTa (Liu et al., 2019), yielding hidden representation H p = [    , h pj ,    ] and H h = [    , h hi ,    ] respectively. As rationalizer is proposed by (Zhao and Vy-diswaran, 2021), we follow this work for rationale identification using cross attention to embed the hypothesis (premise) into premise (hypothesis), which is defined as: a ij = exp (( h hi ) TT anh ( WT 1 h pj )) (cid:80) L p m =0 exp (( h hi ) TT anh ( WT 1 h pm )) (4)  h hi = [ h hi , P ooling ( H p ) , (cid:88) k a ij h pj ] (5) where a ij denotes the attention score of j th token in premise to the i th token in the hypothesis, L p denotes the length of the premise sentence and W 1 is a trainable parameter matrix. The representation of i th token in the hypothesis, denoted as  h hi , is created by concatenating its original state representation, max-pooling representation over h p , and the corresponding sum of attention representation from h p . At last, we use a softmax layer with a linear transformation to model the probability of the i th token in S h being a rationale token. As we have introduced above, counterfactual examples of other classes are of key importance to generate contrastive explanations. In this part, we describe how to generate counterfactual examples. Given a trained neural network model f , the problem of generating counterfactual example for an instance x is to find a set of examples c 1 , c 2 , ..., c k that lead to a desired prediction y (cid:48) . The counterfactual examples are explainable and contrastive when they appropriately consider proximity, diversity and validity. Here, we define a three-part loss function to select qualified counterfactual example: L = L valid +  1 L dist +  2 L div (6) H tokens P tokens Cross Attention H p H h Rationales Token original input P-H pairs Pre-trained LM (RoBERTa / BERT) Reversal Mechanism Loss Evaluation Counterfactual Example Candidate label y and the other label y' Knowledge Extraction from ConcepetNet WHY A Generator (GPT-2) Contrastive Explanation (WHY A NOT B) WHY NOT B Generator (GPT-2) Rationale Identification Counterfactual Example Generation Knowledge-aware Contrastive Explanation Generator where  1 and  2 are hyperparameters for balancing L dist and L div . For generating counterfactual example, the validity term, which ensures the generated counterfactual examples have desired prediction target, is defined as: L valid = k (cid:88) i =1 loss ( f ( c i ) , y (cid:48) ) (7) Meanwhile, the generated examples should be proximal to the original instance as described in (Cheng et al., 2020), which means only a small change needs to be made. We do not expect a big change that transforms a large portion of the original, in which way there will be no difference with merely presenting an example of counter classes and the corresponding explanation will be uninformative or useless. That is, we expect that resultant examples are able to preserve the main content of input while changing domain-related parts. where t indicates a rationale. we calculate the pairwise distance of a set of counterfactual examples and minimize: After defining the loss function, we use a reversal mechanism to produce counterfactual examples. In the reversal mechanism, we use hypernym and hyponym of tokens in WordNet 2 for perturbation. For example, as shown in Figure 2, the original premise and hypothesis are a woman and a young child are making sculptures out of clay and a man and a woman painting on canvas, and the label is contradiction. We find from WordNet the hypernyms of making sculptures out of clay and painting on canvas as doing art and making something respectively. We replace them with their hypernyms to obtain counterfactual examples, and use the model f trained on the original P-H training dataset to predict the resultant examples (Equation 7), and keep those belong to neutral or entailment. After the validity justification, we perform further selection by following Equation 8 and Equation 10, and choose the samples with the smallest loss for neutral and entailment for latter contrastive explanation generation. After obtaining qualified counterfactual examples, some work (Cheng et al., 2020; Wachter et al., 2017; Verma et al., 2020) provides them as counterfactual explanation directly. However, since counterfactual examples do not provide explanations explicitly, it could be difficult for users to understand. Hence, in this part, we focus on generating contrastive explanation via knowledge-aware generative language model, which explain WHY A NOT B rather than merely WHY A. While traditional approach generate explanation with SHAP 3 or LIME 4 , recent work has exploited to use pre-trained generative language models (Radford et al., 2019; Lewis et al., 2020; Raffel et al., 2020). In this paper, we use knowledge-aware pre-trained language model to generate contrastive explanation. Knowledge Extraction Given selected counterfactual examples and identified rationales, we extract relevant knowledge to enhance the generative language model. We acquire structured knowledge and rationale definitions from ConceptNet 5 and dictionary source 6 separately. For ConceptNet, we extract knowledge with Breadth-First-Search (BFS) algorithm as described in (Ji et al., 2020). For dictionary, we extract the definition of rationales by following (Chen et al., 2020a). After extraction, we concatenate these knowledge for training knowledge-aware explanation generator. Knowledge-Aware Explanation Generator For contrastive explanation generation, we divide the WHY A NOT B problem into two simple question: 1) why the label of the input belong to A, 2) why the label of the input not belong to B. In previous study, (Kumar and Talukdar, 2020) proposed a label-specific explanation generator, which fine-tuned GPT2 independently for each label. However, the generator can only produce explanations for WHY A. For the other part of contrastive explanation, we collect some contrastive explanations annotated by human and use them to fine-tune a WHY NOT B generator. Taking a premise-hypothesis pair x along with the qualified counterfactual example x (cid:48) and extracted knowledge KE as input, which is in the form of (cid:104) s (cid:105) Label (cid:104) s (cid:105) x (cid:104) s (cid:105) x (cid:48) (cid:104) s (cid:105) KE (cid:104) s (cid:105) , our fine-tuned language model generates explanations that support the corresponding label in a WHY A NOT 3 https://github.com/slundberg/shap 4 https://github.com/marcotcr/lime 5 https://github.com/commonsense/conceptnet5/ 6 https://dictionary.cambridge.org/ B way. With these explanations, end-users can observe and understand the difference between original input and counterfactual example explicitly. SNLI & e-SNLI The SNLI dataset (Bowman et al., 2015) is a balanced collection of P-H annotated pairs with labels from { entailment, neutral, contradiction } , which consists of about 550K, 10K and 10K examples for train, development, and test set, respectively 7 . (Camburu et al., 2018) extend the SNLI dataset to e-SNLI 8 with natural language explanations of the ground truth labels. Annotators were asked to highlight words in the premise and hypothesis pairs which could explain the labels and write a natural language explanation using the highlighted words. In this paper, we use the highlighted words for rationale identification and use the natural language explanation to fine-tune the language model based WHY A generator. IMDB The IMDB dataset (Maas et al., 2011) is a movie reviews dataset for sentiment classification. It contains 25,000 training data and 25,000 test data with movie reviews labeled as positive or negative. In this paper, we use IMDB as a out-of-domain dataset to evaluate if counterfactual examples can improve the robustness of our model. We are committed to generate contrastive explanations which can distinguish the predicted label and others at semantic level, hence, BLEU (Pa-pineni et al., 2002) score is not a proper way to measure the quality of explanations. That is, it can be better confirmed by manual evaluation. In this work, we use manual evaluation and case study for contrastive explanations quality evaluation. Meanwhile, we use accuracy to measure the effectiveness of generated contrastive explanations on improving model performance in terms of data augmentation (organized in the form of (cid:104) s (cid:105) CE (cid:104) s (cid:105) P remise (cid:104) s (cid:105) Hypothesis (cid:104) s (cid:105) ). RoBERTa & BERT For sequence labelling during rationale identification, we use RoBERTa-large and BERT-large, which have 24 layers, 16 attention heads and a hidden size of 1024 (355M parameters for RoBERTa-large, 340M parameters for BERT-large). For downstream classifications tasks, a clas-sification layer is added over the hidden state of the first [CLS] token at the last layer. GPT-2 For natural language explanation generation, we use the GPT-2 architecture (Radford et al., 2019). In particular, we use the GPT2-medium model that has 24 layers, 16 attention heads and a hidden size of 1024 (345M parameters). We fine-tuned GPT-2 model with label-specific examples that are integrated with contrastive examples and external knowledge from ConceptNet. inference model that considers recursive architectures in both local inference modeling and inference composition, and incorporates syntactic parsing information. (Zhang et al., 2020) incorporate explicit contextual semantics from pre-trained semantic role labeling and introduce an improved language representation model, Semantics-aware BERT (SemBERT), which is capable of explicitly absorbing contextual semantics with a BERT backbone. CA-MTL (Pilault et al., 2021) is a novel transformer based architecture that consists of a new conditional attention mechanism as well as a set of task conditioned modules that facilitate weight sharing, and achieves the new state-of-art performance on SNLI. ETPA (Camburu et al., 2018) propose Explain-Then-Predict-Attention (ETPA) that generates an explanation and then predicts the label with only the generated explanation. NILE:post-hoc (Kumar and Talukdar, 2020) propose natural language inference over label-specific explanations (NILE). A premise and hypothesis pair is input to label-specific a candidate explanation generator that generates natural language explanations supporting the corresponding label. The generated explanations are then fed into an explanation processor, which predicts labels using evidence presented in these explanations. LIREx-base (Zhao and Vydiswaran, 2021) propose LIREx-base that incorporates both a rationale enabled explanation generator and an instance selector to select only relevant, plausible natural language explanations (NLEs) to augment NLI models and evaluate on the standardized SNLI. For rationale identification, we use RoBERTa-base to extract hidden representations and set the learning rate to 2e-5, dropout to 0.02, batch size to 8 and number of epochs to 10. Meanwhile, we use AdamW (Loshchilov and Hutter, 2018) as the optimizer and adopt cross-entropy loss as the loss function. In the counterfactual example generation part, we build a hypernym and hyponym table, and use hypernym and hyponym of tokens in WordNet for perturbation. In the contrastive explanation generation part, we use GPT-2 as the generative language model for training WHY A generator and WHY NOT B Generator. For generator, we set the learning rate to 5e-5, adam epsilon to 1e-8, length for generation to 100. Explanation Generation for SNLI In Table 1, we present the inputs of our model, the results of our approach that include token-level explanation (rationales), counterfactual example and generated contrastive explanation, compared with manually annotated explanation and generated WHY A explanations by NILE:post-hoc and LIREx-base. Compared with WHY A explanations that are simple and lack essential information, the contrastive explanation contains more information such as making sculptures out of clay is a type of art and making sculptures is different from painting on canvas. As shown in Table 1, we provide not only the contrastive explanation but also the identified rationales and reversed counterfactual example for reference. To quantitatively assess contrastive explanations, we compared our method with LIREx-base and NILE:post-hoc in terms of explanation quality through human evaluation on 100 SNLI test samples. The explanation quality refers to whether an explanation provides enough essential information for a predicted label. As shown in Table 2, contrastive explanations produced by our method have a better quality by obtaining over 2.0% and 9.0% than LIREx-base and NILE:post-hoc . Explanation Enhanced NLI In Table 3, we report the experimental results of our method and other baselines include BERT, SemBERT (Zhang et al., 2020), CA-MTL (Pilault et al., 2021), NILE:post-hoc (Kumar and Talukdar, 2020) and LIREx-base (Zhao and Vydiswaran, 2021) on SNLI. With contrastive explanations, we are able to improve the performance of both BERT-large and RoBERTa-large. Compared with NILE:post-hoc (Kumar and Talukdar, 2020), the same scale BERT-large model with contrastive explanations brings a gain of 0.4% on test, which indicates the knowledge-aware contrastive generator are better than the generator of NILE. Compared with LIREx-base that uses RoBERTa-large (Zhao and Vydiswaran, 2021), the BERT-large model and RoBERTa-large with contrastive explanations bring a gain of 0.3% and 1.0% separately, which suggests contrastive explanations are better than rationale enabled explanation. In general, contrastive explanations can achieve new state-of-art performance and get it closer to human annotation (a gain of 1.1% on BERT-Large). We believe that contrastive explanations contain more helpful information (e.g., relations between rationales, differences between original and counterfactual examples) that can be used to improve model performance. Ablation Study We perform ablation studies with BERT-large on the SNLI dataset to evaluate the impacts of different components employed in our method, and report the results in Table 4. We isolated rationales, counterfactual examples and external knowledge, separately. The model without rationales means we generate contrastive explanations with counterfactual examples generated through randomly replacing tokens and extracted external knowledge. The model without counterfactual examples means we extracted knowledge with given rationales and generate contrastive explanation with them. The model without external knowledge means we generate contrastive explanation only with rationales and counterfactual examples. The model without contrastive explanation actually is the BERT-large baseline in SNLI. We can observe that each component is helpful. Especially, if we remove external knowledge and contrastive explanations, we can see a clear decrease of 0.6% and 0.8%, respectively. It indicates that external knowledge and contrastive explanation generation are the most essential components, while rationales and counterfactual examples affect the performance less. On one hand, the ablation study results show, external knowledge and rationales affect more than counterfactual examples on explanation generation. On the other hand, the results suggest that each component contributes positively, and indicate the importance of knowledge aware contrastive explanations, as we highlighted in the title. ex-Table 4: The accuracy (%) of ablation studies on SNLI. amples of IMDB for out of domain evaluation. As shown in Table 5, we train BERT-base on two different training sets: the original training set TRAINO , and the union of original training examples and generated counterfactual examples TRAINO  C , and evaluate it with two separated dev sets: the original dev set DEVO and the generated counterfactual example dev set DEVC . Experimental results shown that BERT-base model enhanced with counterfactual examples achieves 88.5% and 95.1%, bringing a gain of 11.0% on DEVC while a slight decrease of 1.7% on DEVO . It indicates that counterfactual examples can help to improve the robustness of model for more diversified data distribution. With IMDB evaluation, we demonstrate that counterfactual examples can not only help to generate contrastive explanation, but also contribute to data augmentation. In the experiments on SNLI, we evaluated the effectiveness of counterfactual example in contrastive explanation generation. In IMDB experiments, we further verify the effectiveness of counter-factual examples for data augmentation with only rationales identification and heuristic reversal mechanism. Counterfactual example aims to find a minimal change in data that flips the model's prediction and is used for explanation. (Wachter et al., 2017) first propose the concept of unconditional counterfactual explanations and a framework to generate counterfactual explanations. (Hendricks et al., 2018) first consider the evidence that is discriminative for one class but not present in another class, and learn a model to generate counterfactual explanations for why a model predicts class A instead of B. In this paper, we focus on counterfactual example generation providing contrastive example for natural language inference. For post-hoc explainable NLP system, we can divide explanations into three types: feature-based, example-based and concept-based. For feature-based explanation, (Ribeiro et al., 2016) propose LIME and (Guidotti et al., 2018) extend LIME by fitting a decision tree classifier to approximate the non-linear model. However, there is no guarantee that they are faithful to the original model. For example-based explanation, (Kim et al., 2016) select both prototypes and criticisms from the original data points. (Wachter et al., 2017) propose counterfactual explanations providing alternative perturbations. For concept-based explanation, (Ghorbani et al., 2019) explains model decisions through concepts that are more understandable to human than individual features or characters. In this paper, we integrate counterfactual example and concepts for contrastive explanation generation. For natural language inference, (Bowman et al., 2015) propose SNLI which contains samples of premise and hypothesis pairs with human annotations. In order to provide interpretable and robust explanations for model decisions, (Camburu et al., 2018) extend the SNLI dataset with natural language explanations of the ground truth labels, named e-SNLI. For explanation generation in NLI, (Kumar and Talukdar, 2020) propose NILE, which utilizes label-specific generators to produce labels along with explanation. However, (Zhao and Vy-diswaran, 2021) find NILE do not take into account the variability inherent in human explanation, and propose LIREx which incorporates a rationale enabled explanation generator. In this paper, we consider generating contrastive explanations in NLI. In this paper, we focus on knowledge-aware contrastive explanation generation for NLI. We generate counterfactual examples by changing identified rationales of given instances. Afterwards, we extract concepts knowledge from ConceptNet and dictionary to train knowledge-aware explanation generators. We show that contrastive explanations that specify why a model makes prediction A rather than B can provide more faithful information than other WHY A explanations. Moreover, contrastive explanations can be used for data augmentation to improve the performance and robustness of existing model. The exploration of contrastive explanation in other NLP tasks (i.e. question answering) and better evaluation metrics for explanation will be performed in the future. We thank the anonymous reviewers for their helpful comments on this paper. This work is supported by National Key R&D Program of China (No. 2018AAA0101900), the NSFC projects (No. 62072399, No. U19B2042, No. 61402403), Chinese Knowledge Center for Engineering Sciences and Technology, MoE Engineering Research Center of Digital Library, Alibaba Research Intern Program of Alibaba Group, Alibaba-Zhejiang University Joint Institute of Frontier Technologies, and the Fundamental Research Funds for the Central Universities.
abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain method result objective objective method method objective objective method method objective method other method other abstain method method abstain method abstain method abstain method method method abstain abstain method abstain other method method abstain method method method abstain method method method abstain abstain abstain method abstain method method method method method method method abstain method other abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain result result result abstain abstain abstain result result abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain other other abstain method other other other other other abstain other other other other method method method method result abstain abstain other other