e-SNLI: Natural Language Inference with Natural Language Explanations
In order for machine learning to garner widespread public adoption, models must be able to provide interpretable and robust explanations for their decisions, as well as learn from human-provided explanations at train time. In this work, we extend the Stanford Natural Language Inference dataset with an additional layer of human-annotated natural language explanations of the entailment relations. We further implement models that incorporate these explanations into their training process and output them at test time. We show how our corpus of explanations, which we call e-SNLI, can be used for various goals, such as obtaining full sentence justifications of a model's decisions, improving universal sentence representations and transferring to out-of-domain NLI datasets. Our dataset 1 thus opens up a range of research directions for using natural language explanations, both for improving models and for asserting their trust.
Update after Author Feedback:  Thanks for all of the clarification, especially with regard to the BLEU scores. I think the idea of explicable models is worth pursuing, and this is a decent contribution to showing how one might do that. It is unfortunate that this work shows a huge tradeoff between models that perform at high levels and those that explain well (from 4.1 it seems like we can get good performance, but then can't generate correct explanations very often and from 4.2 we can generate correct explanations more often at the expense of good performance). It also seems disappointing that the BLEU scores in the PREDICT setting are already so close to the inter-annotator agreement even though they are not correct explanations very often; this seems to suggest that we really do need to rely on the percent correct given by human evaluation and that the BLEU scores are not very meaningful. This seems like a bottleneck for this resource being widely adopted. Nonetheless, these findings are a solid contribution and so is the data if others are willing to do human evaluation or work on a new automatic metric for a task like this.  Original Review:  This work augments the SNLI corpus with explanations for each label. They then run several different kinds of experiments to show what can be done with such an augmented datasets. First, they show that it is possible to train a model to predict the label and then generate an explanation based on that label. Second, they show that given a ground truth explanation, the labels can be easily predicted. Third, they show that the model that both predicts and explains (e-SNLI) can provide useful sentence representations. Fourth, they try to transfer e-SNLI to MultiNLI. These seem like the perfect experiments for a very interesting attempt to make deep learning models more interpretable. As the authors mentioned, this line of work has some history with Park et al, Ling et al, and Jansen et al, but I have not seen anything quite like this, especially for a task like SNLI, which does hold potential for transfer learning. It is an especially important task to learn how to interpret if we want to have a better sense of how our models are thinking of logical inference.  Subsection 4.1  I’m confused/concerned by the evaluation of the generated explanations and the resulting low BLEU scores. First of all, why use only BLEU-4? Why not use the same BLUE that is typically used for tasks like neural machine translation? This would make the numbers more interpretable to other NLP researchers. Second, is it really .2 in the same scale that NMT researchers would report 30+ BLEU scores for NMT. The obvious answer is no because it is just BLEU-4, but how do I relate those? .2 BLEU on translation would give complete nonsense, but the generated explanations in Table 2 look quite good. How do I make sense of this as a reader? Does this mean the explanations are coherent but mostly ungrounded in the prediction? If the inter-annotator agreement is truly so low, then how can we ever really evaluate these systems? Will we always have to rely on human judgment in comparing future systems to the one you’ve proposed?  I’m also worried by these numbers: if 99% of labels are consistent with their explanations, why is there such a wide gap between the 84% correct labels and the 35.7% correct explanations? Does this mean that the explanations are usually not explaining at all? They just happen to give the same label if you were to infer from them alone? Based on the discussion in the next section, I’m assuming the ‘them’ in ’35.7% of them’ refer to the 84 out of 100 examples correctly predicted rather than out of all 100 qualitatively examined examples, but that was ambiguous until I read 4.2.  How exactly were partial points awarded to get the 35.7% correct explanations?  You mention it is done in the footnote, but I’d like more detail there if there is going to be work that builds off this in the future. Maybe it would just be less ambiguous to not award these partial points.  Subsection 4.2  The 67.75% is quite low in this section. Using the hypothesis alone can give you a score right around there (Gururangan et al. 2018); that should probably be mentioned.  I like this conclusion here. The GENERATE model can just generate explanations after the fact that don’t have any real incentive to be correct. Perhaps with a higher alpha value (Eq. 1), this would change, but we don’t have those experiments to verify.  4.3 and 4.4 appear to show some empirical validation of the method for transfer learning and universal embeddings.   I think there might be a typo in Table 5’s std deviation for InferSent on MultiNLI.   line 203: “a model on model” seems like a typo  I really like this idea, the experiments, and the example generated explanations, but I have major concerns about evaluation for this kind of task. Even though the language for the generated explanations is surprisingly coherent, it seems like in most cases, the explanations are not actually aligning with the predictions in a meaningful way. What’s worse is that the only way to find out is to manually inspect. This doesn’t seem like a sustainable way to compare which models in the future are better at the explanation part of e-SNLI, which is really the crux of the paper. In short, I’m left feeling like the evaluation setup just isn’t quite there yet for the community to be able to use these ideas. This is a hard problem. I don’t have great ideas for how to fix the evaluation, and I’m definitely looking forward to hearing what the authors have to say about this, especially when it comes to the BLEU evaluation and how I might better interpret those results. For now, the overall is a 5 with a confidence of 4.