Despite alarm over the reliance of machine learning systems on so-called spurious patterns in training data, the term lacks coherent meaning in standard statistical frameworks. However, the language of causality offers clarity: spurious associations are those due to a common cause (confounding) vs direct or indirect effects. In this paper, we focus on NLP, introducing methods and resources for training models insensitive to spurious patterns. Given documents and their initial labels, we task humans with revising each document to accord with a counterfactual target label, asking that the revised documents be internally coherent while avoiding any gratuitous changes. Interestingly, on sentiment analysis and natural language inference tasks, classifiers trained on original data fail on their counterfactually-revised counterparts and vice versa. Classifiers trained on combined datasets perform remarkably well, just shy of those specialized to either domain. While classifiers trained on either original or manipulated data alone are sensitive to spurious features (e.g., mentions of genre), models trained on the combined data are insensitive to this signal. We will publicly release both datasets.
This paper addresses the problem of building models for NLP tasks that are robust against spurious correlations in the data by introducing a human-in-the-loop method: annotators are asked to modify data-points minimally in order to change the label.  They refer to this process as counterfactual augmentation.  The authors apply this method to the IMDB sentiment dataset and to SNLI and show (among other things) that many models cannot generalize from the original dataset to the counterfactually-augmented one.
This contribution is timely and addresses a very important problem that needs to be addressed in order to build more robust NLP systems.
Because, however, of a few limitations, I recommend weak acceptance.
My main hesitation comes from a lack of clarity about the main lesson we have learned.  In particular, if the goal is to use this method to augment the data we use to train NLP systems in order to make them more robust, it seems that the time cost of the process will be prohibitive.  On the other hand, perhaps these methods could be used to identify the kind of spurious correlations that models tend to rely on, which could then be used in a more automated data augmentation process.  If that's the goal, however, a more detailed error analysis would need to be included.
A few small comments:
* There was some analysis of the augmented IMDB dataset, but none of the SNLI dataset.  I would love to see a more detailed investigation of what annotators usually did.  For instance, a reason that hypothesis-only models do well is that certain words are very predictive of certain labels (e.g. "not" and contradiction).  Do people leave the negations in when modifying such examples for entailment or neutrality, thus breaking the simple correspondence?  That's a very simple kind of question; more generally, I'd like to see more analysis of the new dataset.
* The BiLSTM they use is very small (embedding and hidden dimension 50).  Given that BERT is most robust against their manipulation, it would be good to see a more powerful recurrent model for comparison.  It would be easy to use ELMo here, if the main question is about Transformers vs recurrent models.
Some very minor / typographic comments:
* abstract: "with revise" should be "with revising"
* first paragraph page 2: some references to causality literature and definition of spuriousness as common cause
* page 2, "We show that..." I'd break this into two sentences to make it easier to parse.
* Table 3: I would make two columns for each model with accuracy on original versus revised.  With the current table, one has to compare cells in the top half of the table to those in the bottom half of the table, which is quite difficult to do.