Natural language inference requires reasoning about contradictions, negations, and their commonsense implications. Given a simple premise (e.g., I'm mad at you), humans can reason about the varying shades of contradictory statements ranging from straightforward negations (I'm not mad at you) to commonsense contradictions (I'm happy). Moreover, these negated or contradictory statements shift the commonsense implications of the original premise in nontrivial ways. For example, while I'm mad implies I'm unhappy about something, negating the premise ( i.e. , I'm not mad) does not necessarily negate the corresponding commonsense implications. In this paper, we present the first comprehensive study focusing on commonsense implications of negated statements and contradictions. We introduce ANION 1 , a new commonsense knowledge graph with 624K if-then rules focusing on negated and contradictory events. We then present joint generative and discriminative inference models for this new resource, providing novel empirical insights on how logical negations and commonsense contradictions reshape the commonsense implications of their original premises. Humans reason about underlying causes and effects of events described in text. For example, in Figure 1, the event X wears a mask is associated with many causal inferences such as X is seen as responsible, or Others get protected. Hypothesizing and reasoning about commonsense inferences is used for understanding complex situations encountered in everyday life (Sap et al., 2019; Bisk et al., 2020; Bhagavatula et al., 2020; Sakaguchi et al., 2020). This ability eludes AI systems, and has motivated the design of a wealth of 1 Data and code available at https://github.com/ liweijiang/anion X takes his mask off Original Commonsense Contradiction X wears a mask X loses his mask responsible get protected to throw the used mask away to have a mask to breath freely nervous to avoid XX is seen as As a result, others want As a result, X feels As a result, X wants Before, X needed As a result, others want As a result, others then Because X wanted Because X wanted Before, X needed saves money from buying a mask As a result, X then X doesn't wear a mask Logical Negation carefree X is seen as X is seen as Figure 1: Commonsense inferences for the event X wears a mask, its logical negation and commonsense contradiction events, and their associated inferences. commonsense knowledge resources, such as Cyc (Lenat, 1995), ConceptNet (Speer et al., 2017), and ATOMIC (Sap et al., 2020; Hwang et al., 2020), to provide structured reasoning capabilities to AI systems (Lin et al., 2019; Feng et al., 2020). However, reasoning about negated observations remains a challenge (Hossain et al., 2020). While negation is often considered a poorer form of meaning than affirmation 2 (Ackrill, 1963; Horn and Wansing, 2020), negated statements can still imply expressive commonsense inferences. In Figure 1, the negated event X doesn't wear a mask, 2 Following Horn and Wansing (2020), we classify declarative expressions as affirmations or negations/contradictions based on whether they affirm or deny an action or object. is connected to rich commonsense inferences, despite describing the absence of action. However, negated observations are rarely found in commonsense knowledge resources. For example, negated examples make up only  3% of examples in the ConceptNet knowledge graph (Li et al., 2016). This scarcity poses downstream issues for systems that must understand negated situations. Commonsense knowledge models (Bosselut et al., 2019; Hwang et al., 2020) trained on resources of largely affirmative instances struggle particularly with negation examples. Their ability to hypothesize inferences for negated events is 35% lower than for affirmative events (4.2). Furthermore, since negated statements are asymmetrically mentioned in text compared to affirmative statements (Jowett et al., 1892; Horn and Wansing, 2020), large-scale pretraining does not implicitly learn negation scoping (Kim et al., 2019). As a result, when presented with negated concepts, pretrained neural language models (PLMs) often exhibit the same associations as affirmative statements (Kassner et al., 2020). Motivated by these observations, our work focuses on improving the ability of knowledge models to make commonsense inferences about events that convey denial, rejection or contradiction of actions. We define our contributions as follows. First, we crowdsource a new large scale resource, A rray of commonse N se I nferences for O ppositions and N egations (ANION ), which contains inferences for different types of negated events.This new resource can be used to train knowledge models on commonsense inferences associated with the absence of actions. Second, we propose a new class of negation discriminators that can be applied to generated commonsense inferences. These discriminators partition inferences based on logical consistency, thereby mitigating the effects of common affirmative associations that violate negation constraints. Discriminators are trained using contrastive samples from paired affirmative and negated events in ANION . Finally, we conduct an empirical study of both of these techniques and show that using trainingand discriminator-based approaches for modeling negation cuts the performance difference between affirmative and negated events by 73% -85% depending on the negation variety. statements into affirmation and negation, which respectively affirms or denies observations about an event (Ackrill, 1963). Despite this seeming simplicity, natural language often expresses negation in complex and subtle ways, using diverse syntactic, semantic and pragmatic formulations (Horn and Wansing, 2020). For example, syntactically, different negation determiners ( i.e. , negation cues) such as no , few and only result in distinct explicit and implicit negative perceptions (Xiang et al., 2016). Despite their diversity, however, negated language expressions are much less likely to appear in text than affirmative statements (Reitan et al., 2015). Consequently, PLMs, which rely on large-scale textual corpora as training data, are prone to decreased performance when confronted with negated constructions. In machine translation, for example, the presence of negation may heavily affect the quality of produced translations (Fancellu and Webber, 2015; Hossain et al., 2020). In factual knowledge understanding tasks, PLMs memorize positive and negative sentences seen during training, but generalize more poorly to unseen negated instances (Kassner and Schtze, 2020). Negation in Commonsense Reasoning Understanding negation and oppositional expressions is critical for reasoning about commonsense knowledge, particularly in counterfactual scenarios (Qin et al., 2019). However, negation is rarely explicitly modeled in NLP studies on commonsense reasoning. As a result, in many NLP tasks, these models experience a performance drop when presented with examples exhibiting negated characteristics. As a case study, the ATOMIC (Sap et al., 2020) knowledge graph encodes social commonsense knowledge about event pre-conditions, event post-conditions, and static attributes in the form of natural language if-then rules. However, despite the fact that ATOMIC provides a rich set of seed events, it comprises an unbalanced set of affirmative events (97.9%) and negated events (2.1%). As a result, when systems link to ATOMIC to retrieve relevant social commonsense inferences, they are likely to recover inferences of affirmative events even when searching for negated instances. Furthermore, knowledge models that use this resource ( e.g. , COMET; Bosselut et al., 2019) are unlikely to learn implicit differences between inferences of affirmative and negated events. When given negated events, these models often produce associations of counterpart affirmative events. For example, for Types Example Negation Cues Example Sentences Affixes un-, ir-, non-, il-, im-, -less, etc. the negated event, X opposes racism, COMET infers X intends to be a racist, an association of the affirmative statement, X supports racism. At the heart of this problem is that inferring commonsense knowledge about negations often requires implicit reasoning. In factual knowledge reasoning, applying logical rules over statements can be effective for handling negative queries (Asai and Hajishirzi, 2020; Ren and Leskovec, 2020). However, directly manipulating affirmative forms with logic-guided rules may fail for commonsense reasoning: the boundary of commonsense inferences between affirmative and negated statements is not always wholly contrastive. Many inferences can be relevant to both forms. The events X puts the potato in the oven and X doesn't put the potato in the oven, could both have an associated inference: X wants to make dinner. The affirmative event clearly implies this inference. For the negated event to be worth mentioning on its own (Grice et al., 1975), an implicit complementary event ( e.g. , X puts the potato in the microwave) would likely hold, which might validate the inference w.r.t. the negated event. To model the defeasibility of commonsense reasoning (Pratt, 1994; Rudinger et al., 2020), modeling both common and contrastive inferences of negated forms is necessary. To provide a rich resource of commonsense inferences for opposition and negation events, we design ANION . Using the same schema as the ATOMIC knowledge graph (Sap et al., 2020), we initialize 22,483 negated forms paired to original ATOMIC events and crowdsource 627,042 new inferences for these negated events. Consistent with ATOMIC , ANION is constructed using English formulations of events and inferences. We briefly recap ATOMIC and describe the construction of ANION below. ATOMIC Background The ATOMIC knowledge graph contains  24K base events ( e.g. , X plays the piano) with 877K accompanying social commonsense inferences ( e.g. , Before, X needs to buy a piano.) along nine dimensions ( e.g. , xNeed ). The full description of ATOMIC relation types can be found in Table 12 in the Appendix. Our knowledge construction pipeline consists of two steps. First, we collect negated and contradictive events by deriving oppositions of events in ATOMIC . Inspired by the distinction made between negation contributed by semantic assertion (explicit negation) or non-asserted content (implicit negation) (Xiang et al., 2016), we define three varieties of negated events: logical negations, semi-logical negations, and commonsense contradictions, which we describe in detail below. Logical and semilogical negations were heuristically formulated from ATOMIC events. Commonsense contradiction events were crowdsourced from Amazon Mechanical Turk (MTurk). Negated events in ANION are assigned to the same data split as the corresponding affirmative event from which they are derived ( e.g. , negated events for ATOMIC training set events are found in the ANION training set). Once a list of negated events is compiled, we crowdsource inferences of these new events on MTurk using similar annotation templates as Sap et al. (2020). We design qualifying tasks to filter out unreliable workers and screen their answers manually for quality control purposes. Logical Negation We define logical negation events as events with the negation cue not added to their original formulation ( e.g. , X does not play the piano). However, different positions of the not modifier in a clause can result in different negation scopes , which can alter the semantics of the event (Councill et al., 2010). To be consistent, we systematically insert not after the subject of the event clause. If necessary, we change verb forms and add auxiliary words ( e.g. , do, does, did, is, was, can, could, would, should, may, might). For quality control, we have human workers validate each logically negated event form and exclude events that annotators identify as uninterpretable or awkwardly worded. For each created event, we then collect the same nine dimensions of inferences as defined in ATOMIC . Consequently, we collected 8,285 logically negated events with 225K corresponding inferences (as shown in Table 2). Appendix A.1 provides further details of the compilation of logical negation events. Semi-logical Negation We define semi-logical negation using explicit cues other than not . We categorize these negation cues (words or phrases) into four subtypes: affixes ( e.g. , legal/illegal), single-word cues ( e.g. , never), multi-word cues ( e.g. , no longer), and negative verbs ( e.g. , refuse). See Table 1 for examples. We create semi-logical negation events by heuristically adding these cues to different positions of ATOMIC events. Similar to logically-negated events, we avoid grammatically incorrect or semantically awkward events by removing auto-generated instances of low quality. The final set of data includes 5,019 semilogical negation events. We then crowdsource a total of 138K inferences for these new events. Appendix A.1 provides further details of the compilation of semi-logical negation events. Commonsense Contradiction We formulate commonsense contradiction as contradictory statements without negation cues. Commonsense contradiction events are not identifiable as negations on their own, but demonstrate reversed semantic or pragmatic meaning when paired with their affirmative counterparts ( e.g. , X eats a hamburger vs. X eats a salad). To obtain commonsense contradictions, we crowdsource two oppositional events for each ATOMIC event, excluding events with blank placeholders representing generic objects, resulting in 40K new commonsense contradiction events. For 9,179 of these events, we crowdsource an additional 262K commonsense inferences. Appendix A.1 provides further details of the crowdsourcing of commonsense contradiction events. ANION can be used as training data for commonsense models to make inferences about negated events. Here, we recap COMET (Bosselut et al., 2019), a commonsense knowledge model, and evaluate how training knowledge models on ANION affects their ability to hypothesize commonsense knowledge for negated and oppositional events. Commonsense transformers (COMET) are generative knowledge models that learn to hypothesize commonsense inferences by training on examples from a knowledge graph. Specifically, COMET receives knowledge tuples in { h, r, t } form during training, where h is a head entity, r is a relation type, and t is a tail entity. The model is trained to maximize the conditional loglikelihood of predicting the tokens of the tail entity t given the tokens of the head entity h and relation r : LG =  (cid:88) log P ( t | h, r ) (1) In ATOMIC and ANION , h corresponds to events, such as X has a nightmare, t corresponds to commonsense inferences about those events, such as X wakes up, and r corresponds to commonsense inference types, such as As a result, X does.... Following Bosselut et al. (2019) and Sap et al. (2020), for each event and relation type in ATOMIC , 10 candidate inferences are decoded from COMET using beam search with b =10. As oppositional instances remain challenging to knowledge models such as COMET, we evaluate how ANION can be used to augment the type of examples seen by COMET during training. Evaluation Metrics Following Bosselut et al. (2019), we evaluate the quality of generated inferences using BLEU-2 (Papineni et al., 2002) as an automatic evaluation. We also compute the perplexity of models on their reference generations. For the human evaluation, we employ human judges from MTurk to identify whether generated commonsense inferences are plausible. We randomly sample 100 events from the original ATOMIC test set along with their negated counterparts from ANION . For each event, we present every decoded inference to five crowdworkers and ask them to identify whether the inference is plausible given the event. For each model trained on a different combination of ATOMIC and ANION ( i.e. , ANION-L, ANION-S, ANION-C), we evaluate the same events for comparison. We calculate Precision @ 10 (P@10) across these human ratings, i.e. , the average number of correct options per event-relation prompt. Specifically, we average the results from 45K ratings to compute the final human score (100 events  9 relations  10 options  5 annotators). The pairwise agreement score of human evaluation is 63.6, which is on par with other similar commonsense reasoning annotation tasks (Rashkin et al., 2016). Does negated event training improve commonsense inference for negated situations? We train a COMET model on the events from ATOMIC ( i.e. , COMET-ATOMIC ), and another on the examples from both ATOMIC and ANION ( i.e. , COMET-FULL ). The combined dataset is shuffled so that the original and negated examples are uniformly mixed during training. We report our comparison of these two models in Table 4. The performance of the original COMET model trained only on the ATOMIC knowledge graph drops significantly across all types of oppositional instances. Most surprisingly, a drop in performance is also observed on commonsense contradictions (ANION-C), which have no explicit negation cues. However, commonsense contradiction events can often be richer in content (see Table 3), making them more challenging for knowledge models. Meanwhile training on all negated examples in the ANION knowledge graph produces significant improvements across all negation categories (ANION -{L,S,C}), though we do observe a slight drop in human ratings on the examples from the original ATOMIC test set. Does negated event training deteriorate commonsense inference of affirmative situations? We note in Table 4 that training on ATOMIC + ANION hurts inference performance on the original ATOMIC evaluation set. To analyze why COMET-FULL does not improve on this set of examples, we perform a case study on inferences generated by COMET-ATOMIC and COMET-FULL under the same event and relation prompt, and note two qualitative patterns. First, we observe that COMET-FULL tends to generate inferences that are less generic, but that may require additional implicit context. For example, for the event X is really sad and the relation xEffect ( i.e. , the effect of the event on X), COMET-ATOMIC generates inferences such as cries, gets depressed and takes medication. Conversely, COMET-FULL generates context-specific inferences such as thinks about the past and thinks about what they did, which, while plausible in some context, may be less straightforward when evaluated broadly (not all feelings of sadness lead to reflection on the past or one's own actions). Second, we find an overall improvement for certain compositional events in ATOMIC that contain conjunction words: and or but. On these examples, COMET-FULL outperforms COMET-ATOMIC with 12.41 and 12.22 BLEU-2 scores respectively. For example, for the event X is hot and humid and the relation xEffect , COMET-ATOMIC 's generation includes correct inferences, such as to take a shower, to cool down, to drink some water, to go outside, and incorrect inferences, such as to turn on the heat and to drink a hot tea. COMET-FULL generates all of COMET-ATOMIC 's correct inferences, but none of the incorrect inferences, demonstrating that training COMET jointly on ATOMIC and ANION can help avoid incorrect inferences involving commonsense mismatch in more compositional situations. In summary, the ability to generate richer, contextual inferences for COMET-FULL is beneficial when handling complex events, but may not be necessary for many of the simple events in ATOMIC , and may backfire when subtler inferences are made. Which variety of negated events are most crucial to include in training sets? As ablations, we train additional models using different subsets of ANION : logical negations (ATOMIC + ANION-L), semi-logical negations (ATOMIC + ANION-S), and commonsense contradictions (ATOMIC + ANION-C). These ablations evaluate whether knowledge models can adapt to certain types of negation more efficiently with additional data. on the evaluation set related to that negation type. Interestingly, though, training on certain types of negation examples can also yield benefits downstream on other negation types. For example, training on commonsense contradictions (ANION-C) provides a clear benefit when evaluating on semi-logically negated events (ANION-S) as opposed to merely training on ATOMIC . Notably, the knowledge model trained with logically negated examples (ATOMIC + ANION-L) outperforms the model trained only on ATOMIC on all test sets. While training on examples of negated events helps knowledge models generate commonsense inferences for these event types, there is still a large gap compared to their performance on affirmative events. To address this discrepancy, we introduce a discriminator-based approach for distinguishing inconsistent inferences of negated events. Our inference discriminator learns to identify plausible and invalid inferences of events by learning from contrastive samples from ATOMIC and ANION . We fine-tune the RoBERTa-base model (Liu et al., 2019) as a binary classifier to identify whether a given knowledge tuple { h, r, t } is logically valid. The model is trained on paired original and negated events as described below. Such training examples inject implicit commonsense nuances that differ between oppositional events to teach the discriminator to identify logical pitfalls. Training details for discriminators can be found in Appendix A.3. Data The paired events used to train the negation discriminator are automatically constructed from the ATOMIC and ANION knowledge graphs. Positive examples can be constructed by sampling tuples from each knowledge graph. To construct negative training samples, we introduce the concept of common and contrast sets among inferences of events and their oppositions. Common and contrast sets distinguish how commonsense inferences are not necessarily negated in the same manner as their corresponding events. While certain inferences of events are also in opposition to a negated event, some may be common. For the events X eats a cheeseburger and X eats a salad, an inference such as X is hungry might be common to both events while inferences such as X is unhealthy or X is healthy would be viewed as contrastive. Specifically, we assume two head events in ATOMIC and ANION , and their respective set of tail inferences regarding a common relation type. We define the common set of these inferences as the intersection of the two sets of tail inferences connected to each head event by applying the exact match of string forms. The contrast set is formed by distinct tail inferences connected to the two head events. Logically valid ( i.e. , positive) training examples consist of knowledge tuples from ATOMIC and ANION . Logically invalid ( i.e. , negative) training examples are formed by swapping the set of contrast set inferences between paired original and negated events. 3 To balance the training set, we sample the same number of positive and negative tuples for original and negation events. Statistics of the resulting training sets are in Table 6. Using different portions of ANION for training yields four unique discriminators ( i.e. , L , S , C and 3 We note that annotations in ATOMIC and ANION are finite ( i.e. , not covering the full space of possible commonsense inferences about events). As a result, it is possible that in a more expansive annotation, elements of the contrast sets would in fact be part of the common set of an event and its negation. For the purpose of this work, however, contrast sets were an efficient way of acquiring high-quality semantically negative examples for training discriminators. LSC ) that we apply to commonsense inferences generated by COMET. The discriminators classify each option as either logically valid or invalid , partitioning the candidates into two sets, which we evaluate with human judgements. As a baseline, we also record the precision of not using a discriminator, which assumes all generated inferences are valid candidates ( i.e. , the all set). Metrics We evaluate and compare the quality of the all , valid and invalid sets using BLEU-2 and the same human evaluation as in 4. The all set contains the full set of 10 candidates, while the valid and invalid sets have varying number of elements depending on how discriminators classify them, summing to 10. To compute statistical significance between valid and all sets, we use a permutation test with 100K permutations. Details are provided in Appendix A.4. Do discriminators effectively distinguish inconsistent inferences? The results in Table 7 demonstrate that the discriminator trained on all subsets of ANION ( LSC ) can select subsets of inferences ( i.e. , Event + Rel Generation V PX does not skate around xAttr athletic (cid:55) (cid:55) careless (cid:55) (cid:55) lazy (cid:51) (cid:51) uncoordinated (cid:51) (cid:51) unskilled (cid:51) (cid:51) X does not sit behind Y xIntent to be alone (cid:51) (cid:51) to be left alone (cid:51) (cid:51) to avoid Y (cid:51) (cid:51) to sit (cid:55) (cid:55) to wait (cid:51) (cid:55) X does not look angry xNeed to calm down (cid:55) (cid:51) to watch a movie (cid:51) (cid:55) to have been provoked (cid:55) (cid:55) to not be angry (cid:51) (cid:51) to be calm (cid:51) (cid:51) X refuses to hear a scary noise xWant to run away (cid:55) (cid:55) to go to sleep (cid:51) (cid:51) to be safe (cid:51) (cid:51) to keep quiet (cid:51) (cid:51) to avoid the noise (cid:51) (cid:51) X never brings Y into conflicts oWant to avoid X (cid:55) (cid:55) to be left alone (cid:55) (cid:51) to thank X (cid:51) (cid:51) to fight back (cid:55) (cid:55) to avoid conflict (cid:55) (cid:51) X scarcely gets sunburned xReact burned (cid:55) (cid:55) hurt (cid:55) (cid:55) sick (cid:55) (cid:55) sad (cid:55) (cid:55) satisfied (cid:51) (cid:51) X under no circumstancesforgetsY'swallet oReact upset (cid:55) (cid:55) sad (cid:55) (cid:55) angry (cid:55) (cid:55) thankful (cid:51) (cid:51) grateful (cid:51) (cid:51) X has trouble with advertising X's business xEffect loses money (cid:51) (cid:51) loses clients (cid:51) (cid:51) gets fired (cid:51) (cid:51) gets sued (cid:55) (cid:55) cries (cid:51) (cid:51) X puts Y out of mind oEffect has a better day (cid:55) (cid:55) becomes sad (cid:51) (cid:51) cries (cid:51) (cid:51) is grateful towards X (cid:55) (cid:55) feels better (cid:55) (cid:55) Table 8: Inferences of randomly selected ANION events by COMET-ATOMIC . the valid set) that are more logically consistent with their seed event. This observation holds across all evaluation subsets of ANION , as well as the original ATOMIC evaluation set. Table 8 shows examples of valid and invalid candidates for negated and contradicted events from ANION as specified by the LSC discriminator. The discriminator is notably Eval Disc L S C LSCATOMIC all 55.69 55.93 56.94 58.30 valid 55.65 56.18 57.26 59.07 %iprv -0.07 0.44 0.57 1.32 ANION-L all 39.46 37.85 36.43 39.45 valid **46.39 **41.93 37.54 **45.59 %iprv 17.55 10.78 3.03 15.57 ANION-S all 37.13 39.29 37.72 38.55 valid 37.48 **44.58 39.03 **44.93 %iprv 0.96 13.47 3.45 16.56 ANION-C all 46.92 47.32 48.26 48.81 valid 46.83 47.68 48.79 *51.45 %iprv -0.20 0.75 1.09 5.40 Table 9: P@{# valid} scores of the all and valid sets determined by the L , S , C and LSC discriminators. good at identifying invalid inferences wrongly associated to corresponding affirmative events ( e.g. , athletic and careless for the event X does not skate around under the relation, xAttr ). However, this analysis leaves open the possibility that we are generating too many inferences for each event, but that the decoder could rank correct inferences higher among the full set of generated candidates. To evaluate this possibility, we count the number of elements in the valid sets for each example and only keep the same number of the top-scoring elements from the all set (scored using generation perplexity). In Table 9, we see the average precision score for the pruned all sets (P@{# valid}) still underperforms the precision of their corresponding valid sets. to provide a discriminator for? To examine the generalization effects of each negation type, we also train discriminators on a single negation subset of ANION examples ( i.e. , L , S , C ) and compare the P@{# valid} score of the all and valid sets. Results in Table 9 indicate that each discriminator is best for identifying valid inferences for the types of events on which it was trained. The L , S , and C discriminators all achieve improvements when partitioning events similar to their training. However, the LSC discriminator trained on all negation forms shows the largest valid set improvement across all discriminators on ATOMIC , ANION-S, and ANION-C. On ANION-L, the LSC discrimina-Eval Disc L S C LSCATOMIC all 54.16 54.49 55.03 55.68 valid 54.20 54.64 55.71 **57.58 %iprv 0.08 0.28 1.23 3.41 ANION-L all 46.54 46.26 46.15 46.39 valid **50.71 **48.36 46.16 **49.85 %iprv 8.98 4.54 0.03 7.45 ANION-S all 46.90 47.73 47.47 47.53 valid 47.14 **50.42 48.20 **50.62 %iprv 0.51 5.65 1.55 6.50 ANION-C all 50.80 51.29 51.28 51.83 valid 50.94 51.52 52.65 *53.91 %iprv 0.28 0.45 2.67 4.02 Table 10: P@{# valid} scores of the all and valid sets determined by the L , S , C and LSC discriminators. tor still yields a significantly improved valid set. Are learning-based and discriminator-based approaches complementary? We apply our discriminators to the generations of the COMET model trained on ANION . In Table 10, we see that the LSC discriminator, when applied to generations of COMET trained on ANION , achieves significant improvements over all evaluation sets, including the original events. The full evaluation of the P@{# valid} and P@3 scores of applying different discriminators to generations of COMET trained on different data over all evaluation sets are shown in Table 13 and 14 in Appendix A. Can discriminators be used to more aggressively generate inferences? While applying discriminators to generated inferences yields a valid subset with higher accuracy, we are left with fewer correct inferences in total. Thus, we investigate the efficiency of using discriminators to expand the number of inferences generated. We decode inferences from COMET with beam size 25, and then apply the discriminator to this larger candidate set. Table 11 shows that for logical negation, the valid set of beam 25 has higher accuracy and more correct options than the all set of beam 10. Thus, when we have a larger and potentially more noisy set of candidates, applying the negation discriminator yields a set of options that have higher quality than using all the candidates from a smaller set of initial generations. We present the first comprehensive study on commonsense implications of negations and contradictions. To expand commonsense resources for the challenge of negation modeling, we introduce ANION , a large scale commonsense knowledge graph for negated and contradicted events. We use ANION to train commonsense knowledge models and demonstrate that it effectively enriches machine commonsense inference capabilities around negation. Lastly, we propose a negation discriminator capable of identifying logical flaws in commonsense inferences. By combining the model trained on ANION with the negation discriminator, we achieve a further performance boost. We select English as the base language of ANION so that our resource may be directly linked with the original ATOMIC knowledge graph. We acknowledge, however, that resources in English are more likely to reflect the mindsets and behaviors of English speakers. Furthermore, and in our case specifically, our annotators were primarily from the US. Consequently, this language choice biases the content of the knowledge graph toward North American perspectives, which affects what models trained on these resources would learn about social norms (Acharya et al., 2021). Future works may also include other languages and cultures to make the ANION resource more culturally and ideologically inclusive. We recruit crowdworkers from MTurk who are located within the US with HIT approval rates higher than 98%. To ensure high quality task completions, we post pilot batches and manually examine tens of thousands of responses to identify users who provide high quality annotations. We select 834 qualified users for the formal data collection and human evaluation tasks. Since the entire study spans multiple months, we regularly sample responses to re-examine their quality during the formal study, and remove HITs from crowdworkers who provide decreased-quality responses over time. We are particularly cautious about the human evaluation tasks, so even with qualified users, we still comprehensively examine tens of thousands of human evaluation tasks by grouping HITs per users, and look at their responses together to identify potential spamming behaviors and inconsistencies. For the data collection and human evaluation tasks, we aimed to compensate crowdworkers with an average of $15 per hour. To ensure a fair payment, we first post a pilot task to evaluate average time cost of a specific task, and pay users at a high rate in this round to avoid underpayment during the pilot study. We then calculate new payment from the pilot task such that approximately 75% of the HITs would have been paid with more than $15 per hour at the adjusted rate in the pilot round. We then adopt this new rate for the formal study. We repeat the above procedure of determining payment periodically during the study to ensure the crowdworkers are consistently well-paid. The authors thank Elisa Kreiss for helpful discussions. We also thank the anonymous reviewers and meta-reviewers for their helpful feedback. This research was supported in part by DARPA under the MCS program through NIWC Pacific (N66001-19-2-4031), and the Allen Institute for AI (AI2).
abstain abstain abstain abstain objective objective objective abstain abstain abstain other abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain objective objective objective objective abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain objective method abstain abstain method method method result abstain method objective abstain method method abstain objective abstain method abstain objective abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain other abstain abstain abstain result abstain method abstain abstain abstain other abstain abstain method result abstain method abstain method abstain method objective abstain objective objective result abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain abstain abstain abstain other other other