Many natural language questions require qualitative, quantitative or logical comparisons between two entities or events. This paper addresses the problem of improving the accuracy and consistency of responses to comparison questions by integrating logic rules and neural models. Our method leverages logical and linguistic knowledge to augment labeled training data and then uses a consistency-based regularizer to train the model. Improving the global consistency of predictions, our approach achieves large improvements over previous methods in a variety of question answering (QA) tasks including multiple-choice qualitative reasoning, cause-effect reasoning, and extractive machine reading comprehension. In particular, our method significantly improves the performance of RoBERTa-based models by 1-5% across datasets. We advance state of the art by around 5-8% on WIQA and QuaRel and reduce consistency violations by 58% on HotpotQA. We further demonstrate that our approach can learn effectively from limited data. 1 1 Introduction Comparison-type questions (Tandon et al., 2019; Tafjord et al., 2019; Yang et al., 2018) ask about relationships between properties of entities or events such as cause-effect, qualitative or quantitative reasoning. To create comparison questions that require inferential knowledge and reasoning ability, annotators need to understand context presented in multiple paragraphs or carefully ground a question to the given situation. This makes it challenging to annotate a large number of comparison questions. Most current datasets on comparison questions are much smaller than standard machine reading comprehension (MRC) datasets (Rajpurkar 1 Our code and data is available at https://github. com/AkariAsai/logic_guided_qa . Q: The ceramic vase was less flexible than the plastic ball so it was A: more breakable Q: The ceramic vase was more flexible than the plastic ball so it was A: less breakable Q: If it is silent, does the outer ear collect less sound waves? A: more [positive causal relationship] Q: If the outer ear collect less sound waves, is less sound being detected? A : more [positive causal relationship] Q: If it is silent, is less sound being detected? A: more [positive causal relationship] RoBERTa more breakable more breakable RoBERTa more more less Conflict Conflict Figure 1: Inconsistent predictions by RoBERTa. Top row shows an example of symmetric inconsistency and the second row shows an example of transitive inconsistency. The examples are partially modified. et al., 2016; Joshi et al., 2017). This poses new challenges to standard models, which are known to exploit statistical patterns or annotation artifacts in these datasets (Sugawara et al., 2018; Min et al., 2019a). Importantly, state-of-the-art models show inconsistent comparison predictions as shown in Figure 1. Improving the consistency of predictions has been previously studied in natural language inference (NLI) tasks (Minervini and Riedel, 2018; Li et al., 2019), but has not been addressed in QA. In this paper, we address the task of producing globally consistent and accurate predictions for comparison questions leveraging logical and symbolic knowledge for data augmentation and training regularization. Our data augmentation uses a set of logical and linguistic knowledge to develop additional consistent labeled training data. Subsequently, our method uses symbolic logic to incorporate consistency regularization for additional supervision signal beyond inductive bias given by data augmentation. Our method generalizes previous consistency-promoting methods for NLI tasks (Minervini and Riedel, 2018; Li et al., 2019) to adapt to substantially different question formats. Our experiments show significant improvement over the state of the art on a variety of QA tasks: a classification-based causal reasoning QA, a multiple choice QA for qualitative reasoning and an extractive MRC task with comparisons between entities. Notably, our data augmentation and consistency constrained training regularization improves performance of RoBERTa-based models (Liu et al., 2019) by 1.0%, 5.0% and 2.5% on WIQA, QuaRel and HotpotQA. Our approach advances the state-of-the-art results on WIQA and QuaRel with 4.7 and 8.4% absolute accuracy improvement, respectively, reducing inconsistent predictions. We further demonstrate that our approach can learn effectively from limited labeled data: given only 20% of the original labeled data, our method achieves performance on par with a competitive baseline learned with the full labeled data. Data augmentation has been explored in a variety of tasks and domains (Krizhevsky et al., 2009; Cubuk et al., 2019; Park et al., 2019). In NLP, using back-translation (Yu et al., 2018) or dictionary based word replacement (Zhang et al., 2015) has been studied. Most relevant to our work, Kang et al. (2018) study NLI-specific logic and knowledge-based data augmentation. Concurrent to our work, Gokhale et al. (2020) study visual QA models' ability to answer logically composed questions, and show the effectiveness of logic-guided data augmentation. Our data augmentation does not rely on task-specific assumptions, and can be adapted to different formats of QA task. We further leverage consistency-promoting regularization, which gives improvements in accuracy and consistency. Improving prediction consistency via training regularization has been studied in NLI tasks. Minervini and Riedel (2018) present model-dependent first-order logic guided adversarial example generation and regularization. Li et al. (2019) introduce consistency-based regularization incorporating the first-order logic rules. Previous approach is model-dependent or relies on NLI-specific rules, while our method is model-agnostic and is more generally applicable by combining it with data augmentation. Regularizing loss to penalize violations of structural constraints in models' output has been also studied in previous work on constraint satisfaction in structured learning (Lee et al., 2019; Ganchev et al., 2010). Our work regularizes models to produce globally consistent predictions among augmented data following logical constraints, while those studies incorporates structured prediction models following linguistics rules. We present the components of our QA method: first-order logic guided data augmentation (Sec-tion 3.1 and Section 3.2), and consistency-based regularization (Section 3.3). For globally consistent predictions in QA, we require responses to follow two important general logical rules: symmetric consistency and transitive consistency , which are illustrated in Figure 1 and are formally described below. Let q, p, a be a question, a paragraph and an answer predicted by a model. A is a set of answer candidates. Each element of A can be a span in p , a class category, or an arbitrary answer choice. X = { q, p, a } represents a logic atom. Symmetric consistency In a comparison question, small surface variations such as replacing words with their antonyms can reverse the answer, while keeping the overall semantics of the question as before. We define symmetry of questions in the context of QA as follows: ( q, p, a  )  ( q sym , p, a  sym ) , where q and q sym are antonyms of each other, and a  sym is the opposite of the ground-truth answer a  in A . For example, the two questions in the first row of Figure 1 are symmetric pairs. We define the symmetric consistency of predictions in QA as the following logic rule: ( q, p, a )  ( q sym , p, a sym ) , (1) which indicates a system should predict a sym given ( q sym , p ) , if it predicts a for ( q, p ) . Transitive consistency. Transitive inference between three predicates A, B, C is represented as: A  B  B  C then A  C (Gazes et al., 2012). In the context of QA, the transitive examples are mainly for causal reasoning questions that inquire about the effect e given the cause c . The second row of Figure 1 shows an example where transitive consistency is violated. For two questions q 1 and q 2 in which the effect of q 1 ( = e 1 ) is equal to the cause of q 2 ( = c 2 ), we define the transitive consistency of predictions as follows: ( q 1 , p, a 1 )  ( q 2 , p, a 2 )  ( q trans , p, a trans ) . Given a set of training examples X in the form of ( q, p, a  ) , we automatically generate additional examples X aug = { q aug , p, a  aug } using symmetry and transitivity logical rules. The goal is to augment the training data so that symmetric and transitive examples are observed during training. We provide some augmented examples in Table 1. Augmenting symmetric examples To create a symmetric question, we convert a question into an opposite one using the following operations: (a) replace words with their antonyms, (b) add, or (c) remove words. For (a), we select top frequent adjectives or verbs with polarity (e.g., smaller, increases ) from training corpora, and expert annotators write antonyms for each of the frequent words (we denote this small dictionary as D ). More details can be seen in Appendix A. For (b) and (c), we add negation words or remove negation words (e.g., not ). For all of the questions in training data, if a question includes a word in D for the operation (a), or matches a template (e.g., which * is  which * is not ) for operations (b) and (c), we apply the operation to generate q sym . 2 We obtain a  sym by re-labeling the answer a  to its opposite answer choice in A (see Appendix B). Augmenting transitive examples We first find a pair of two cause-effect questions X 1 = ( q 1 , p, a  1 ) and X 2 = ( q 2 , p, a  2 ) , whose q 1 and q 2 consist of 2 We observe that (b)(c) are less effective than (a) in WIQA or QuaRel, while especially (b) contributes to the performance improvements on HotpotQA as much as (a) does. ( c 1 , e 1 ) and ( c 2 , e 2 ) , where e 1 = c 2 holds. When a  1 is a positive causal relationship , we create a new example X trans = ( q 3 , p, a  2 ) for q 3 = ( c 1 , e 2 ) . Sampling augmented data Adding all consistent examples may change the data distribution from the original one, which may lead to a deterioration in performance (Xie et al., 2019). One can select the data based on a model's prediction inconsistencies (Minervini and Riedel, 2018) or randomly sample at each epoch (Kang et al., 2018). In this work, we randomly sample augmented data at the beginning of training, and use the same examples for all epochs during training. Despite its simplicity, this yields competitive or even better performance than other sampling strategies. 3 3.3 Logic-guided Consistency Regularization We regularize the learning objective (task loss, L task ) with a regularization term that promotes consistency of predictions (consistency loss, L cons ). The first term L task penalizes making incorrect predictions. The second term L cons 4 penalizes making predictions that violate symmetric and transitive logical rules as follows: L cons =  sym L sym +  trans L trans , (4) where  sym and  trans are weighting scalars to balance the two consistency-promoting objectives. 3 We do not add X aug if the same pair has already exist. 4 We mask the L cons for the examples without symmetric or transitive consistent examples. Previous studies focusing on NLI consistency (Li et al., 2019) calculate the prediction inconsistency between a pair of examples by swapping the premise and the hypothesis, which cannot be directly applied to QA tasks. Instead, our method leverages consistency with data augmentation to create paired examples based on general logic rules. This enables the application of consistency regularization to a variety of QA tasks. Inconsistency losses The loss computes the dissimilarity between the predicted probability for the original labeled answer and the one for the augmented data defined as follows: L sym = | log p ( a | q, p )  log p ( a aug | q aug , p ) | . Likewise, for transitive loss, we use absolute loss with the product T-norm which projects a logical conjunction operation ( q 1 , p, a 1 )  ( q 2 , c, a 2 ) to a product of probabilities of two operations, p ( a 1 | q 1 , p ) p ( a 2 | q 2 , p ) , following Li et al. (2019). We calculate a transitive consistency loss as: L trans = | log p ( a 1 | q 1 , p ) + log p ( a 2 | q 2 , p )  log p ( a trans | q trans , p ) | . Annealing The model's predictions may not be accurate enough at the beginning of training for consistency regularization to be effective. We perform annealing (Kirkpatrick et al., 1983; Li et al., 2019; Du et al., 2019). We first set  { sym,trans } = 0 in Eq. (4) and train a model for  epochs, and then train it with the full objective. Datasets and experimental settings We experiment on three QA datasets: WIQA (Tandon et al., 2019), QuaRel (Tafjord et al., 2019) and HotpotQA (oracle, comparison questions 5 ) (Yang et al., 2018). As shown in Table 1, these three datasets are substantially different from each other in terms of required reasoning ability and task format. In WIQA, there are 3,238 symmetric examples and 4,287 transitive examples, while 50,732 symmetric pairs and 1,609 transitive triples are missed from the original training data. HotpotQA and QuaRel do not have any training pairs requiring consistency. Our method randomly samples 50, 80, 90% of the augmented data for WIQA, QuaRel and HotpotQA, resulting in 24,715/836/3,538 newly created training examples for those datasets, respectively. We use standard F1 and EM scores for performance evaluation on HotpotQA and use accuracy for WIQA and QuaRel. We report a violation of consistency following Minervini and Riedel (2018) to evaluate the effectiveness of our approach for improving prediction consistencies. We compute the violation of consistency metric v as the percentage of examples that do not agree with symmetric and transitive logical rules. More model and experimental details are in Appendix. Main Results Table 2 demonstrates that our methods ( DA and DA + Reg ) constantly give 1 to 5 points improvements over the state-of-the-art RoBERTa QA's performance on all three of the datasets, advancing the state-of-the-art scores on WIQA and QuaRel by 4.7% and 8.4%, respectively. On all three datasets, our method signifi-WIQA Input RoBERTa DA DA+Reg p Sound enters the ears of a person. cantly reduces the inconsistencies in predictions, demonstrating the effects of both data augmentation and regularization components. Notably on WIQA, RoBERTa shows violation of consistency in 13.9% of the symmetric examples and 10.0% of the transitive examples. Our approach reduces the violations of symmetric and transitive consistencies to 8.3% and 2.5%, respectively. Results with limited training data Table 2 also shows that our approach is especially effective under the scarce training data setting: when only 20% of labeled data is available, our DA and Reg together gives more than 12% and 14% absolute accuracy improvements over the RoBERTa baselines on WIQA and QuaRel, respectively. Ablation study We analyze the effectiveness of each component on Table 3. DA and Reg each improves the baselines, and the combination performs the best on WIQA and QuaRel. DA (standard) follows a previous standard data augmentation technique that paraphrases words (verbs and adjectives) using linguistic knowledge, namely Word-Net (Miller, 1995), and does not incorporate logical rules. Importantly, DA (standard) does not give notable improvement over the baseline model both in accuracy and consistency, which suggests that logic-guided augmentation gives additional inductive bias for consistent QA beyond amplifying the number of train data. As WIQA consists of some transitive or symmetric examples, we also report the performance with Reg only on WIQA. The performance improvements is smaller, demonstrating the importance of combining with DA . Qualitative Analysis Table 4 shows qualitative examples, comparing our method with RoBERTa baseline. Our qualitative analysis shows that DA+Reg reduces the confusion between opposite choices, and assigns larger probabilities to the ground-truth labels for the questions where DA shows relatively small probability differences. On HotpotQA, the baseline model shows large consistency violations as shown in Table 2. The HotpotQA example in Table 4 shows that RoBERTa selects the same answer to both q and q sym , while DA answers correctly to both questions, demonstrating its robustness to surface variations. We hypothesize that the baseline model exploits statistical pattern, or dataset bias presented in questions and that our method reduces the model's tendency to exploit those spurious statistical patterns (He et al., 2019; Elkahky et al., 2018), which leads to large improvements in consistency. We introduce a logic guided data augmentation and consistency-based regularization framework for accurate and globally consistent QA, especially under limited training data setting. Our approach significantly improves the state-of-the-art models across three substantially different QA datasets. Notably, our approach advances the state-of-the-art on QuaRel and WIQA, two standard benchmarks requiring rich logical and language understanding. We further show that our approach can effectively learn from extremely limited training data. This research was supported by ONR N00014-18-1-2826, DARPA N66001-19-2-403, NSF (IIS1616112, IIS1252835), Allen Distinguished Investigator Award, Sloan Fellowship, and The Naka-jima Foundation Fellowship. We thank Antoine Bosselut, Tim Dettmers, Rik Koncel-Kedziorski, Sewon Min, Keisuke Sakaguchi, David Wadden, Yizhong Wang, the members of UW NLP group and AI2, and the anonymous reviewers for their insightful feedback.
abstain result objective result result method objective abstain abstain abstain other abstain abstain abstain method objective method abstain result result result objective other other abstain abstain abstain abstain other other other method other method method method abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain method method abstain method method abstain abstain abstain abstain method abstain abstain method abstain abstain abstain method method method abstain abstain abstain abstain abstain method abstain abstain method abstain method abstain abstain method method abstain method abstain abstain method method abstain method method abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain result abstain result other other