Deep NLP models have been shown to be brittle to input perturbations. Recent work has shown that data augmentation using counterfactuals  i.e. minimally perturbed inputs  can help ameliorate this weakness. We focus on the task of creating counterfactuals for question answering, which presents unique challenges related to world knowledge, semantic diversity, and answerability. To address these challenges, we develop a R etrieveG enerate-F ilter (RGF) technique to create counterfactual evaluation and training data with minimal human supervision. Using an open-domain QA framework and question generation model trained on original task data, we create counterfactuals that are fluent, semantically diverse, and automatically labeled. Data augmentation with RGF counterfactuals improves performance on out-of-domain and challenging evaluation sets over and above existing methods, in both the reading comprehension and open-domain QA settings. Moreover, we find that RGF data leads to significant improvements to robustness to local perturbations. 1 1 Introduction Models for natural language understanding (NLU) may outperform humans on standard benchmarks, yet still often perform poorly under a multitude of distributional shifts (Jia and Liang (2017); Naik et al. (2018); McCoy et al. (2019), inter alia ) due to over-reliance on spurious correlations or dataset artifacts. This behavior can be probed using counterfactual data (Kaushik et al., 2020; Gardner et al., 2020) designed to simulate interventions on specific attributes: for example, perturbing the movie review A real stinker, one out of ten!" to A real classic, ten out of ten!" allows us to discern the  Work performed during an internship at Google. 1 Code at https://github.com/ google-research/language/tree/master/language/qa_counterfactuals RETRIEVE ( REALM) GENERATEFILTER Wikipedia Who is the captain of the Richmond Football Club? List of Richmond Football Club captains >> Jeff Hogg 1994 -1996 ... Current Captain: Trent Cotchin The ... Richmond Football club ran a women's team and ... Jess Kennedy was named the team ' s captain Who captained Richmond Football Club's women's team? Who won the inaugural best in VFL 2018 season? Who captained Richmond Football Club's women's team? Figure 1: Retrieve-Generate-Filter to generate counterfactual queries for Natural Question (Kwiatkowski et al., 2019) using an open-domain retrieval system, question generation and post-hoc filtering. effect of adjective polarity on the model's prediction. Many recent works (Kaushik et al., 2020, 2021; Wu et al., 2021a; Geva et al., 2021, inter alia ) have shown that training augmented with this counterfactual data (CDA) improves out-of-domain generalization and robustness against spurious correlations. Consequently, several techniques have been proposed for the automatic generation of counterfactual data for several downstream tasks (Wu et al., 2021a; Ross et al., 2021b,a; Bitton et al., 2021; Geva et al., 2021; Asai and Hajishirzi, 2020; Mille et al., 2021). In this paper, we focus on counterfactual data for question answering, in both the reading comprehension and open-domain settings (e.g. Rajpurkar et al., 2016; Kwiatkowski et al., 2019). Model inputs consist of a question and optionally a context passage, and the target a is a short answer span. Counterfactuals are often considered in the context of a specific causal model (Miller, 2019; Halpern and Pearl, 2005), but in this work we follow Wu et al. (2021a) and Kaushik et al. (2020) and seek a method to generate counterfactuals that may be use-1670 ful in many different settings. In QA, the set of possible causal features is large and difficult to specify a priori ; relevant factors are often instance-specific and exploring them may require world knowledge. For example, going from Who is the captain of the Richmond Football Club to a perturbed question Who captained Richmond's women's team? as in Figure 1 requires knowledge about the club's alternate teams, and the perturbation Who was the captain of RFC in 1998? requires knowledge about the time-sensitive nature of the original question. In the absence of such knowledge, otherwise reasonable edits  such as Who captained the club in 2050?  can result in false premises or unanswerable questions. We develop a simple yet effective technique to address these challenges: R etrieve, G enerate, and F ilter (RGF; Figure 1). We use the near-misses of a retrieve-and-read QA model to propose alternate contexts and answers which are closely related to  but semantically distinct from  the original question. We then use a sequence-to-sequence question generation model (Alberti et al., 2019) to generate corresponding questions to these passages and answers. This results in fully-labeled examples, which can be used directly to augment training data or filtered post-hoc for analysis. While our method requires no supervised inputs besides the original task training data, it is able to generate highly diverse counterfactuals covering a range of semantic phenomena (4), including many transformation types which existing methods generate through heuristics (Dua et al., 2021), meaning representations (Ross et al., 2021b; Geva et al., 2021) or human generation (Bartolo et al., 2020; Gardner et al., 2020). Compared to alternative sources of synthetic data (5.1), training augmented with RGF data improves performance on a variety of settings (5.2, 5.3), including out-of-domain (Fisch et al., 2019) and contrast evaluation sets (Bartolo et al., 2020; Gardner et al., 2020), while maintaining in-domain accuracy. Additionally, we introduce a measure of pairwise consistency , and show that RGF significantly improves robustness to a range of local perturbations (6). 2 Related Work 2.1 Counterfactual Generation There has been considerable interest in developing challenge sets for NLU that evaluate models on a wide variety of counterfactual scenarios. Gardner et al. (2020); Khashabi et al. (2020); Kaushik et al. (2020); Ribeiro et al. (2020) use humans to create these perturbations, optionally in an adversarial setting against a particular model (Bartolo et al., 2020). However, these methods can be expensive and difficult to scale. This has led to an increased interest in creating automatic counterfactual data for evaluating out-of-distribution generalization (Bowman and Dahl, 2021) and for counterfactual data augmentation (Geva et al., 2021; Longpre et al., 2021). Some work focuses on using heuristics like swapping superlatives and nouns (Dua et al., 2021), changing gendered words (Webster et al., 2020), or targeting specific data splits (Finegan-Dollak and Verma, 2020). More recent work has focused on using meaning representation frameworks and structured control codes (Wu et al., 2021a), including grammar formalisms (Li et al., 2020), semantic role labeling (Ross et al., 2021b), structured image representations like scene graphs (Bitton et al., 2021), and query decompositions in multi-hop reasoning datasets (Geva et al., 2021). Ye et al. (2021) and Longpre et al. (2021) perturb contexts instead of questions by swapping out all mentions of a named entity. The change in label can be derived heuristically or requires a round of human re-labeling of the data. These may also be difficult to apply to tasks like Natural Questions (Kwiatkowski et al., 2019), where pre-defined schemas can have diffi-culty covering the range of semantic perturbations that may be of interest. 2.2 Data Augmentation Non-counterfactual data augmentation methods for QA, where the synthetic examples are not paired with the original data, have shown only weak improvements to robustness and out-of-domain generalization (Bartolo et al., 2021; Lewis et al., 2021). Counterfactual data augmentation is hypothesized to perform better, as exposing the model to minimal pairs should reduce spurious correlations and make the model more likely to learn the correct, causal features (Kaushik et al., 2020). However, Joshi and He (2021) find that methods that limit the structural and semantic space of perturbations can potentially hurt generalization to other types of transformations. This problem is exacerbated in the question answering scenario where there can be multiple semantic dimensions to edit. Our method attempts to address this by targeting a broad range 1671 of semantic phenomena, thus reducing the chance for the augmented model to overfit. 3 RGF: Counterfactuals for Information-seeking Queries We define a counterfactual example as an alternative input x (cid:48) which differs in some meaningful, controlled way from the original x , which in turn allows us to reason  or teach the model  about changes in the label (the outcome). For question-answering, we take as input triples ( q, c, a ) consisting of the question, context passage, and short answer, and produce counterfactual triples ( q (cid:48) , c (cid:48) , a (cid:48) ) where a (cid:48) (cid:54) = a . This setting poses some unique challenges, such as the need for background knowledge to identify relevant semantic variables to alter, ensuring sufficient semantic diversity in question edits, and avoiding questions with false premises or no viable answers. Ensuring (or characterizing) minimality can also be a challenge, as small changes to surface form can lead to significant semantic changes, and vice-versa. We introduce a general paradigm for data generation  R etrieve, G enerate and F ilter  to tackle these challenges. 3.1 Overview of RGF An outline of the RGF method is given in Figure 1. Given an input example x = ( q, c, a ) consisting of a question, a context paragraph, and the corresponding answer, RGF generates a set of new examples N ( x ) = { ( q (cid:48) 1 , c (cid:48) 1 , a (cid:48) 1 ) , ( q (cid:48) 2 , c (cid:48) 2 , a (cid:48) 2 ) , . . . } from the local neighborhood around x. We first use an open-domain retrieve-and-read model to retrieve alternate contexts c (cid:48) and answers a (cid:48) where a (cid:54) = a (cid:48) . As near-misses for a task model, these candidates ( c (cid:48) , a (cid:48) ) are closely related to the original target ( c, a ) but often differ along interesting, latent semantic dimensions (Figure 2) in their relation to the original question, context, and answer. We then use a sequence-to-sequence model to generate new questions q (cid:48) from the context and answer candidates ( c (cid:48) , a (cid:48) ) . This yields triples ( q (cid:48) , c (cid:48) , a (cid:48) ) which are fully labeled, avoiding the problem of unanswerable or false-premise questions. Compared to methods that rely on a curated set of minimal edits (e.g. Wu et al., 2021b; Ross et al., 2021b), our method admits the use of alternative contexts 2 c (cid:48) (cid:54) = c , and we do not explicitly constrain 2 An alternative approach would be to make direct, targeted edits to the original context c . However, beyond a limited space of local substitutions (Longpre et al., 2021; Ye et al., our triples to be minimal perturbations during the generation step. Instead, we use post-hoc filtering to reduce noise, select minimal candidates, or select for specific semantic phenomena based on the relation between q and q (cid:48) . This allows us to explore a significantly more diverse set of counterfactual questions q (cid:48) (C.1), capturing relations that may not be represented in the original context c . We describe each component of RGF below; additional implementation details are provided in Appendix A. 3.2 Retrieval We use REALM retrieve-and-read model of (Guu et al., 2020). REALM consists of a BERT-based bi-encoder for dense retrieval, a dense index of Wikipedia passages, and a BERT-based answer-span extraction model for reading comprehension, all fine-tuned on Natural Questions (NQ; Kwiatkowski et al., 2019). Given a question q , REALM outputs a ranked list of contexts and answers within those contexts: { ( c (cid:48) 1 , a (cid:48) 1 ) , ( c (cid:48) 2 , a (cid:48) 2 ) , . . . ( c (cid:48) k , a (cid:48) k ) } . These alternate contexts and answers provide relevant yet diverse background information to construct counterfactual questions. For instance, in Figure 1, the question Who is the captain of the Richmond Football Club" with answer Trent Cotchin" also returns other contexts with alternate answers like Jeff Hogg" ( q (cid:48) = Who captained the team in 1994" ), and Steve Morris" ( q (cid:48) = Who captained the reserve team in the VFL league" ). Retrieved contexts can also capture information about closely related or ambiguous entities. For instance, the question who wrote the treasure of the sierra madre" retrieves passages about the original book Sierra Madre , its movie adaptation, and a battle fought in the Sierra de las Cruces mountains. This background knowledge allows us to perform contextual-ized counterfactual generation, without needing to specify a priori the type of perturbation or semantic dimension. To focus on label-transforming counterfactuals, we retain all ( c (cid:48) i , a (cid:48) i ) where a (cid:48) i does not match any of the gold answers a from the original NQ example. 3.3 Question Generation This component generates questions q (cid:48) that correspond to the answer-context pairs ( c (cid:48) , a (cid:48) ) . We use a T5 (Raffel et al., 2020) model fine-tuned 2021; Ross et al., 2021a) this is very difficult due to the need to model complex discourse and knowledge relations. 1672 on ( q, c, a ) triples from Natural Questions, using context passages as input with the answer marked with special tokens. We use the trained model to generate questions ( q (cid:48) 1 , q (cid:48) 2 , . . . q (cid:48) k ) for each of the the retrieved set of alternate contexts and answers, (( c (cid:48) 1 , a (cid:48) 1 ) , ( c (cid:48) 2 , a (cid:48) 2 ) , . . . ( c (cid:48) k , a (cid:48) k )) . For each ( c (cid:48) i , a (cid:48) i ) , we use beam decoding to generate 15 different questions q (cid:48) . We measure the fluency and correctness of generated questions in 4. 3.4 Filtering for Data Augmentation Noise Filtering The question generation model can be noisy, resulting in a question that cannot be answered given c (cid:48) or for which a (cid:48) is an incorrect answer. Round-trip consistency (Alberti et al., 2019; Fang et al., 2020) uses an existing QA model to answer the generated questions, ensuring that the predicted answer is consistent with the target answer provided to the question generator. We use an ensemble of six T5-based reading-comprehension ( ( q, c )  a ) models, trained on NQ using different random seeds (Appendix A), and keep any generated ( q (cid:48) , c (cid:48) , a (cid:48) ) triples where at least 5 of the 6 models agree on the answer. This discards about 5% of the generated data, although some noise still remains; see 4 for further discussion. Filtering for Minimality Unlike prior work on generating counterfactual perturbations, we do not explicitly control for the type of semantic shift or perturbation in the generated questions. Instead, we use post-hoc filtering over generated questions q (cid:48) to encourage minimality of perturbation. We define a filtering function f ( q, q (cid:48) ) that categorizes the semantic shift or perturbation in q (cid:48) with respect to q . One simple version of f is the word-level edit (Levenshtein) distance between q and q (cid:48) . After noise filtering, for each original ( q, c, a ) triple we select the generated ( q (cid:48) , c (cid:48) , a (cid:48) ) with the smallest non-zero word-edit distance between q and q (cid:48) such that a (cid:54) = a (cid:48) . We use this simple heuristic to create large-scale counterfactual training data for augmentation experiments (5). Over-generating potential counterfactuals based on latent dimensions identified in retrieval and using a simple filtering heuristic avoids biasing the model toward a narrow set of perturbation types (Joshi and He, 2021). 3.5 Semantic Filtering for Evaluation To better understand the types of counterfactuals generated by RGF, we can apply additional filters based on question meaning representations to cat-Question from NQ Original: who is the captain of richmond football club? Predicate: who is the captain of X? Reference Change CF1: who is the captain of richmond's vfl reserve team? Predicate: who is the captain of X? Predicate Change CF2: who wears number 9 for richmond football club? Predicate: who wears Y for X? Predicate and Reference Change CF3: who did graham negate in the grand final last year? Predicate: who did X negate in Y last year? Table 1: Categorization of generated questions based on QED decomposition. The original reference Richmond football Club" changes in CF1 and CF3. Predicate Who is the captain" changes in CF2 and CF3. egorize counterfactual ( q, q (cid:48) ) pairs for evaluation. Meaning representations provide a way to decompose a question into semantic units and categorize ( q, q (cid:48) ) based on which of these units are perturbed. In this work, we employ the QED formalism for explanations in question answering (Lamm et al., 2021). QED decompositions segment the question into a predicate template and a set of reference phrases. For example, the question Who is captain of richmond football club" decomposes into one question reference richmond football club" and the predicate Who is captain of X" . A few example questions and their QED decompositions are illustrated in Table 1. We use these question decompositions to identify the relation between a counterfactual pair ( q, q (cid:48) ) . Concretely, we fine-tune a T5-based model on the QED dataset to perform explanation generation following the recipe of Lamm et al. (2021), and use this to identify predicates and references for the question from each ( q, c, a ) triple. We use exact match between strings to identify reference changes. As predicates can often differ slightly in phrasing ( who captained vs. who is captain ), we take a predicate match to be a prefix matching with more than 10 characters. For instance, Who is the captain of Richmond's first ever women's team?" , Who is the captain of the Richmond Football Club" have same predicates. We filter generated questions into three perturbation categories  reference change, predicate change, or both. 4 Intrinsic Evaluation Following desiderata from Wu et al. (2021a) and Ross et al. (2021b), we evaluate our RGF data 1673 Player Specific Game outcome Who has won the women's single winbledon tennis tournament in 2018 Who won the women's singles Australian Open? Who won the women's doubles at Wimbledon 2015 How many games in Wimbledon final set tie break? Who won the Wimbledon women's singles title in 2016 Who won the runner's up in the women's singles at Wimbledon in 2018 Who did Serena Williams best in the Wimbledon finals 2015 GameType Misc. TournamentName Tournamentyear Locative Country what's the population of walnut grove minnesota? what's the population of walnut grove washington? what is the population of apple valley minnesota ? how many students at walnut grove secondary school ? how long has the walnut twig beetle been in california ? where is walnut grove located in minnesota ? what is the population of walnut grove bc ? TownName Population based StateName Misc Who won the men's singles at wimbledon? along three measures: fluency , correctness , and directionality . Fluency Fluency measures whether the generated text is grammatically correct and semantically meaningful. Fluency is very high from RGF, as the generation step leverages a high-quality pretrained langauge model (T5). We manually annotate a subset of 100 generated questions, and find that 96% of these are fluent. Correctness Correctness measures if the generated question q (cid:48) and context, alternate answer pairs ( c (cid:48) , a (cid:48) ) are aligned, i.e. the question is answerable given context c (cid:48) and a (cid:48) is that answer. We quantify correctness in the generated dataset by manually annotating a samples of 100 ( q (cid:48) , c (cid:48) , a (cid:48) ) triples (see Appendix B). The proportion of noise varies from 30% before noise filtering and 25% after noise filtering using an ensemble of models (3.4). Directionality/Semantic Diversity In Table 2, we show examples of semantic changes that occur in our data, including reference changes (50% of changes), predicate changes (30%), negations (1%), question expansions, disambiguations, and contractions (13%). These cover many of the transformations found in prior work (Gardner et al., 2020; Ross et al., 2021b; Min et al., 2020b), but RGF is able to achieve these without the use of heuristic transformations or structured meaning representations. As shown in Figure 2, the types of relations are semantically rich and cover attributes relevant to each particular instance that would be difficult to capture with a globally-specified schema. Additional examples are shown in Figure 6. 1674 Exact Match (RC) Train Size NQ SQuAD TriviaQA HotpotQA BioASQ AQA AmbigQA Original NQ 90K 70.91 80.26 13.67 50.57 35.90 27.00 46.81 Ensemble 90K 71.29 80.50 13.86 50.57 36.90 27.80 46.90 Gold Agen-Qgen 90K + 90K 70.80 67.71 10.83 42.69 30.63 19.40 41.95 Rand. Unlike many counterfactual generation methods, RGF natively creates fully-labeled ( q (cid:48) , c (cid:48) , a (cid:48) ) examples which can be used directly for counterfactual data augmentation (CDA). We augment the original NQ training set with additional examples from RGF, shuffling all examples in training. We explore two experimental settings, reading comprehension (5.2) and open-domain QA (5.3), and compare RGF-augmented models to those trained only on NQ, as well as to alternative baselines for synthetic data generation. As described in Section 3.4, we use edit-distance based filtering to choose one generated ( q (cid:48) , c (cid:48) , a (cid:48) ) triple to augment for every original example, ( q, c, a ) . 3 Additional training details for all models and baselines are included in Appendix A. 5.1 Baselines In the abstract, our model for generating counterfactuals specifies a way of selecting contexts c (cid:48) from original questions, and answers a (cid:48) within those contexts, and a way of a generating questions q (cid:48) from them. RGF uses a retrieval model to identify relevant contexts; here we experiment with two baselines that use alternate ways to select c (cid:48) . We also compare to the ensemble of six reading comprehension models described in 3.4, with answers selected by majority vote. Random Passage (Rand. Agen-Qgen) Here, c (cid:48) is a randomly chosen paragraph from the Wikipedia index, with no explicit relation with the original question. This setting simulates generation from the original data distribution of Natural Questions. To ensure that the random sampling of Wikipedia paragraphs has a similar distribution, we employ the learned passage selection model from Lewis 3 We don't see significant gains from adding more data beyond this; see Appendix C.3 et al. (2021), 4 . This baseline corresponds to the model of Bartolo et al. (2021), which was applied to the SQuAD dataset (Rajpurkar et al., 2016); our version is trained on NQ and omits AdversarialQA. Gold Context (Gold Agen-Qgen) Here, c (cid:48) is the passage c containing the original short answer a from the NQ training set. This baseline specifically ablates the retrieval component of RGF, testing whether the use of alternate passages leads to more diversity in the resulting counterfactual questions. Answer Generation for Baselines For both the above baselines for context selection, we select spans in the new passage that are likely to be answers for a potential counterfactual question. We use a T5 (Raffel et al., 2020) model fine-tuned for question-independent answer selection c  a on NQ, and select the top 15 candidates from beam search. To avoid simply repeating the original question, we only retain answer candidates a (cid:48) which do not match the original NQ answers a for that example. These alternate generated answer candidates and associated passages are then used for question generation and filtering as in RGF (3.3). For the Gold Agen-Qgen case, we select based on the longest edit distance between ( q, q (cid:48) ) , which gave significantly better performance than random selection or the shortest edit distance used for RGF. In the reading comprehension (RC) setting, the input consists of the question and context and the task is to identify an answer span in the context. Thus, we augment training with full triples ( q (cid:48) , c (cid:48) , a (cid:48) ) consisting of the retrieved passage c (cid:48) , generated and filtered question q (cid:48) , and alternate answer a (cid:48) . with input consisting of the question prepended to the context. We evaluate domain generalisation of our RC models on three evaluation sets from the MRQA 2019 Challenge (Fisch et al., 2019). We also measure performance on evaluation sets consisting of counterfactual or perturbed versions of RC datasets on Wikipedia, including SQuAD (Ra-jpurkar et al., 2016), AQA (adversarially-generated SQuAD questions; Bartolo et al., 2020), and human authored counterfactual examples (contrast sets; Gardner et al., 2020) from the QUOREF dataset (Dasigi et al., 2019). We also evaluate on the set of disambiguated queries in AmbigQA (Min et al., 2020b), which by construction are minimal edits to queries from the original NQ. Results We report exact-match scores in Table 3; F1 scores follow a similar trend. We observe only limited improvements on the in-domain NQ development set, but we see significant improvements from CDA with RGF data in out-of-domain and challenge-set evaluations compared both to the original NQ model and the Gold and Random baselines. RGF improves by 1-2 EM points on most challenge sets, and up to 7 EM points on the BioASQ set compared to training on NQ only, while baselines often underperform the NQ-only model on these sets. Note that all three augmentation methods have similar proportion of noise (Appendix B), so CDA's benefits may be attributed to improving model's ability to learn more robust features for the task of reading comprehension. Using an ensemble of RC models improves slightly on some tasks, but does not improve on OOD performance as much as RGF. RGF's superior performance compared to the Gold Agen-Qgen baseline is especially interesting, since the latter also generates topically related questions. We observe that RGF counterfactuals are more closely related to the original question compared to this baseline (Fig-ure 5 in Appendix C), since q (cid:48) is derived from a near-miss candidate ( c (cid:48) , a (cid:48) ) to answer the original q (S3.1). In the open-domain (OD) setting, only the question is provided as input. The pair ( q (cid:48) , a (cid:48) ) , consisting of generated and filtered question q (cid:48) and alternate answer a (cid:48) , is used for augmentation. Compared to the RC setting where passages change as well, here the edit distance filtering of 3.4 ensures the augmentation data represents minimal perturbations. Experimental Setting We use the method and implementation from Guu et al. (2020) to finetune REALM on ( q, a ) pairs from NQ. End-to-end training of REALM updates both the reader model and the query-document encoders of the retriever module. We evaluate domain generalization on popular open-domain benchmarks: TriviaQA (Joshi et al., 2017), SQuAD (Rajpurkar et al., 2016), Curated TREC dataset (Min et al., 2021), and disambiguated queries from AmbigQA (Min et al., 2020b). Results In the open-domain setting (Table 4), we observe an improvement of 2 EM points over the original model even in-domain on Natural Questions, while also improving significantly when compared to other data augmentation techniques. RGF improves over the next best baseline  Random Agen-Qgen  by up to 6 EM points (on TriviaQA). We hypothesize that data augmentation has more benefit in this setting, as the open-domain task is more difficult than reading comprehension, and counterfactual queries may help the model learn better query and document representations to improve retrieval. To better understand how CDA improves the model, we introduce a measure of local consistency (6.1) to measure model robustness, and perform a strat-ified analysis (6.2) to show the benefits of the semantic diversity available from RGF. Compared to synthetic data methods such as PAQ (Lewis et al., 2021), RGF generates counterfactual examples that are paired with the original inputs and concentrated in local neighborhoods around them (Figure 2). As such, we hypothesize that augmentation with this data should specifically improve local consistency, i.e. how the model behaves under small perturbations of the input. Experimental Setting We explicitly measure how well a model's local behavior respects perturbations to input. Specifically, if a model f : ( q, c )  a correctly answers q , how often does it also correctly answer q (cid:48) ? We define pairwise consistency as accuracy over the counterfactuals ( q (cid:48) , a (cid:48) , c (cid:48) ) , conditioned on correct predictions for the original examples: C ( D ) = ED [ f ( q (cid:48) , c (cid:48) ) = a (cid:48) | f ( q, c ) = a ] 1676 Exact Match (OD) Train Size NQ TriviaQA AmbigQA SQuAD v1.0 TREC Original 90K 37.65 26.75 22.43 14.25 31.93 Gold Agen-Qgen 90K + 90K 37.86 27.02 23.65 15.01 32.94 Rand. To measure consistency, we construct validation sets consisting of paired examples ( q, c, a ) , ( q (cid:48) , c (cid:48) , a (cid:48) ) : one original, and one counterfactual. We use QED to categorize our data, as described in 3.5. Specifically, we create two types of pairs: (a) a change in reference where question predicate remains fixed, and (b) a change in predicate, where the original reference(s) are preserved. 5 We create a clean evaluation set by first selecting RGF examples for predicate or reference change, then manually filtering the data to discard incorrect triples (4) until we have 1000 evaluation pairs of each type (see Appendix B). We also construct paired versions of AQA, AmbigQA, and the QUOREF contrast set. For AmbigQA, we pair two disambiguated questions and for QUOREF, we pair original and human-authored counterfactuals. AQA consists of human-authored adversarial questions q (cid:48) which are not explicitly paired with original questions; we create pairs by randomly selecting an original question q and a generated question q (cid:48) from the same passage. Results Training with RGF data improves consistency by 12-14 points on the QED-filtered slices of RGF data, and 5-7 points on AQA, AmbigQA and QUOREF contrast (Table 5). The Gold Agen-Qgen baseline (which contains topically related queries about the same passage) also improves consistency over the original model compared to the Random Agen-Qgen baseline or to the ensemble model, though not by as much as RGF. Consistency improvements on AQA, AmbigQA and QUOREF are especially noteworthy, since they suggest an improvement in robustness to local perturbations that is independent of other confounding distributional similarities between training and evaluation data. QED-based decomposition of queries allows for the creation of label-changing counterfactuals along orthogonal dimensions  a change of reference or predicate. We investigate whether training towards one type of change induces generalization bias, a detrimental effect which has been observed in tasks such as NLI (Joshi and He, 2021). have the same reference (predicate change) or same predicate (reference change), as defined in 3.5. We over-generate by starting with 20 ( q (cid:48) , c (cid:48) , a (cid:48) ) for each original training example to ensure that we find at least one q (cid:48) that matches the criterion. We also evaluate on paired evaluation sets from 6.1. Results Results are shown for QED-filtered training in Table 5. Counterfactual perturbation of a specific kind (a predicate or a reference change) during augmentation does not hurt performance on another perturbation type compared to the baseline NQ model, which differs from the observations of Joshi and He (2021) on NLI. Furthermore, similar to the observations of Min et al. (2020a), augmenting with one type of perturbation has orthogonal benefits that improve model generalization on another perturbation type: augmenting with RGF (  Pred.) leads to significant improvement on RGF (  Ref.), and vice-versa. Compared to reference-change examples, augmenting with predicate-change examples leads to greater improvements in local consistency, except for on RGF (  Ref.) and on AmbigQA  which contains many reference-change pairs. Predicate-change examples may also be more informative to the model, as reference changes can be modeled more easily by lexical matching within common context patterns. Joshi and He (2021) show CDA to be most effective in the low-resource regime. To better understand the role that dataset size plays in CDA in the reading comprehension setting, we evaluate RGF in a cross-domain setting where only a small amount of training data is available. Experimental Setting Since our approach depends on using an open-domain QA model and a question generation model trained on all Natural Questions data, we instead experiment with a low-resource transfer setting on the BioASQ domain, which consists of questions on the biomedical domain. We use the domain-targeted retrieval model from Ma et al. (2021), where synthetic question-passage relevance pairs generated over the PubMed corpus are used to train domain-specific retrieval without any gold supervised data. We fine-tune our question-generation model on (limited) in-domain data, generate RGF data for augmentation, and then use this along with (limited) in-domain data to further fine-tune an RC model, using the NQ-trained weights for initialization. Results We observe significant improvements over the baseline model in the low resource setting for in-domain data (< 2000 examples), as shown in Table 6. Compared with the limited gains we see on the relatively high-resource NQ reading comprehension task, we find that on BioASQ, CDA with 1000 examples improves performance by 2% F1 and 3% exact match, performing nearly as well as a model trained on 2000 gold examples. These results suggest that using counterfactual data in lieu of collecting additional training data is especially useful in the low-resource setting. Retrieve-Generate-Filter (RGF) creates counterfactual examples for QA which are semantically diverse, using knowledge from the passage context and a retrieval model to capture semantic changes that would be difficult to specify a priori with a global schema. The resulting examples are fully-labeled, and can be used directly for training or filtered using meaning representations for analysis. We show that training with this data leads to improvements on open-domain QA, as well as on challenge sets, and leads to significant improvements in local robustness. While we focus on question answering, for which retrieval components are readily available, we note that the RGF paradigm is quite general and could potentially be applied to other tasks with a suitable retrieval system.
abstain abstain method objective method abstain result abstain other abstain abstain abstain result abstain result abstain result abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain method abstain method method abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain method method method method other abstain abstain abstain result method