Despite significant interest in developing general purpose fact checking models, it is challenging to construct a large-scale fact verification dataset with realistic real-world claims. Existing claims are either authored by crowdworkers, thereby introducing subtle biases that are difficult to control for, or manually verified by professional fact checkers, causing them to be expensive and limited in scale. In this paper, we construct a large-scale challenging fact verification dataset called FAVIQ, consisting of 188k claims derived from an existing corpus of ambiguous information-seeking questions. The ambiguities in the questions enable automatically constructing true and false claims that reflect user confusions (e.g., the year of the movie being filmed vs. being released). Claims in FAVIQ are verified to be natural, contain little lexical bias, and require a complete understanding of the evidence for verification. Our experiments show that the state-of-the-art models are far from solving our new task. Moreover, training on our data helps in professional fact-checking, outperforming models trained on the widely used dataset FEVER or in-domain data by up to 17% abso-lute. Altogether, our data will serve as a challenging benchmark for natural language understanding and support future progress in professional fact checking. 1 1 Introduction Fact verification, the task of verifying the factuality of the natural language claim, is an important NLP application (Cohen et al., 2011) and has also been used to evaluate the amount of external knowledge a model has learned (Petroni et al., 2021). However, it is challenging to construct fact verification data with claims that contain realistic and implicit misinformation. Crowdsourced claims from prior work such as FEVER (Thorne et al., 2018a) are  Equal Contribution 1 Data available at https://faviq.github.io . written with minimal edits to reference sentences, leading to strong lexical biases such as the overuse of explicit negation and unrealistic misinformation that is less likely to occur in real life (Schuster et al., 2019). On the other hand, data constructed by professional fact-checkers are expensive and are typically small-scale (Hanselowski et al., 2019). In this paper, we show it is possible to use information-seeking questions (Kwiatkowski et al., 2019) and their ambiguities (Min et al., 2020) to construct a large-scale, challenging, and realistic fact verification dataset. Information-seeking questions are inherently ambiguous because users do not know the answers to the questions they are posing. For example, in Figure 1, the question is ambiguous because the filming of the movie and the release of the movie can both be seen as the creation time. We introduce a new dataset FAVIQ  FA ct V erification derived from I nformation-seeking Q uestions, which uses such ambiguities to generate challenging fact verification problems. For instance, the claim in Figure 1 requires the model to identify that the movie released in 2001 is in fact filmed in 2000 and to return refute . Like this, claims generated through the crossover of the disambiguation of information-seeking questions 5154 are likely to contain misinformation that real users are easily confused with. We automatically generate such claims by composing valid and invalid question-answer pairs and transforming them into textual claims using a neural model. The data is further augmented by claims from regular question-answer annotations. In total, FAVIQ consists of 188k claims. We manually verified a subset of claims to ensure that they are as natural as human-written claims. Our analysis shows that the claims have significantly lower lexical bias than existing crowdsourced claims; claims involve diverse types of distinct entities, events, or properties that are semantically close, being more realistic and harder to verify without a complete understanding of the evidence text. Our experiments show that a model with no background knowledge performs only slightly better than random guessing, and the state-of-the-art model achieves an accuracy of 65%, leaving significant room for improvement. Furthermore, training on FAVIQ improves the accuracy of verification of claims written by professional fact checkers, outperforming models trained on the target data only or pretrained on FEVER by up to 17% absolute. Together, our experiments demonstrate that FAVIQ is a challenging benchmark as well as a useful resource for professional fact checking. To summarize, our contributions are three-fold: 1. We introduce FAVIQ, a fact verification dataset consisting of 188k claims. By leveraging information-seeking questions and their natural ambiguities, our claims require the identification of entities, events, or properties that are semantically close but distinct, making the fact verification problem very challenging and realistic. 2. Our experiments show that the state-of-the-art fact verification models are far from solving FAVIQ, indicating significant room for improvement. 3. Training on FAVIQ significantly improves the verification of claims written by professional fact checkers, indicating that FAVIQ can support progress in professional fact checking. as a benchmark to evaluate the knowledge in model (Petroni et al., 2021). One line of work has studied professional fact checking, dealing with claims collected by professional fact checkers in specific domains (Vlachos and Riedel, 2014; Ferreira and Vlachos, 2016; Au-genstein et al., 2019; Hanselowski et al., 2019). While such data contains realistic claims that have occurred in the real world, it is expensive to construct as it requires labor from professional fact checkers. Moreover, it is less suitable as a benchmark due to lack of a standard evidence corpus such as Wikipedia 2 and ambiguities in labels. 3 Other fact verification datasets are collected through crowdsourcing (e.g., FEVER (Thorne et al., 2018a) and its variants (Thorne et al., 2018b; Thorne and Vlachos, 2019)) by altering a word or negating the reference text to intentionally make true or false claims. This process leads to large-scale datasets but with strong artifacts and unrealistic claims (Schuster et al., 2019; Thorne and Vlachos, 2019; Eisenschlos et al., 2021). Consequently, a trivial claim-only baseline with no evidence achieves near 80% (Petroni et al. (2021), verified in Section 4.1). While more recent work proposes new crowdsourcing methods that alleviate artifacts (Schuster et al., 2021; Eisenschlos et al., 2021), their claims are still written given particular evidence text, being vulnerable to subtle lexical biases that can be hard to explicitly measure. We construct a fact verification dataset from highly ambiguous information-seeking questions. Our claims have significantly less lexical bias than other crowdsourced ones (Figure 3), contain realistic misinformation that people are likely to be confused about (Table 4), and are challenging to current state-of-the-art models (Section 4.1). Moreover, training a model on our data improves professional fact checking (Section 4.2). QA to Verification Task Prior work has also used QA data to create entailment or fact verification benchmarks. Most make use of synthetic or annotated questions (Demszky et al., 2018; Jiang et al., 2020; Pan et al., 2021; Chen et al., 2021) 4 2 For this reason, prior work on professional fact checking assumes gold evidence document. while we use questions posed by real users to reflect confusions that naturally occur while seeking information. Thorne et al. (2021) use information-seeking questions, by converting yes/no questions to support / refute claims, but at a small scale and with unambiguous questions. Instead, our work uses large-scale information-seeking questions (with no restriction in answers) to claims. We are also unique in using highly ambiguous QA pairs to obtain claims that are more challenging to verify and have significantly fewer lexical cues (quantitative comparisons in Section 3.3). We construct FAVIQ  FA ct V erification derived from I nformation-seeking Q uestions, where the model is given a natural language claim and predicts support or refute with respect to the English Wikipedia. The key idea to construct the data is to gather a set of valid and invalid question-answer pairs (Section 3.1.2) from annotations of information-seeking questions and their ambiguities (Section 3.1.1), and then convert each question-answer pair ( q, a ) to a claim (Section 3.1.3). Figure 2 presents an overview of this process. We use QA data from Natural Questions (NQ, Kwiatkowski et al. (2019)) and AmbigQA (Min et al., 2020). NQ is a large-scale dataset consisting of the English information-seeking questions mined from Google search queries. AmbigQA provides disambiguated question-answer pairs for NQ questions, thereby highlighting the ambiguity that is inherent in information-seeking questions. Given an ambiguous question, it provides a set of multiple distinct answers, each paired with a new disambiguated question that uniquely has that answer. 3.1.2 Composing Valid and Invalid QA Pairs FAVIQ is constructed from ambiguous questions and their disambiguation ( A set) and is further augmented by using unambiguous question-answer pairs ( R set). From ambiguous questions ( A set) We use the data consisting of a set of ( q, { q 1 , a 1 } , { q 2 , a 2 } ) , where q is an information seeking question that has a 1 , a 2 as multiple distinct answers. 5 q 1 and q 2 are disambiguated questions for the answers a 1 and a 2 , i.e., q 1 has a 1 as a valid answer and a 2 as an invalid answer. We use ( q 1 , a 1 ) and ( q 2 , a 2 ) as valid question-answer pairs, and ( q 1 , a 2 ) and ( q 2 , a 1 ) as invalid question-answer pairs. This data is particularly well suited to fact checking because individual examples require identification of entities, events, or properties that are semantically close but distinct: the fact that a user asked an ambiguous question q without realizing the difference between ( q 1 , a 1 ) and ( q 2 , a 2 ) indicates that the distinction is non-trivial and is hard to notice without sufficient background knowledge about the topic of the question. From regular questions ( R set) We use the QA data consisting of a set of ( q, a ) : an information-seeking question q and its answer a . We then obtain an invalid answer to q , denoted as a neg , from an off-the-shelf QA model for which we use the model from Karpukhin et al. (2020)DPR followed by a span extraction model. We choose a neg 5 If q has more than two distinct answers, we sample two. with heuristics to obtain hard negatives but not the false negative; details provided in Appendix A. We use ( q, a ) and ( q, a neg ) as a valid and an invalid question-answer pair, respectively. We can think of ( q, a neg ) as a hard negative pair chosen adversarially from the QA model. 6 This data can be obtained on a much larger scale than the A set because annotating a single valid answer is easier than annotating disambiguations. We transform question-answer pairs to claims by training a neural model which maps ( q, a ) to a claim that is support if and only if a is a valid answer to q , otherwise refute . We first manually convert 250 valid and invalid question-answer pairs obtained through Section 3.1.2 to claims. We then train a T5-3B model (Raffel et al., 2020), using 150 claims for training and 100 claims for validation. The model is additionally pretrained on data from Demszky et al. (2018), see Appendix A. 3.1.4 Obtaining silver evidence passages We obtain silver evidence passages for FAVIQ by (1) taking the question that was the source of the claim during the data creation (either a user question from NQ or a disambiguated question from AmbigQA), (2) using it as a query for TF-IDF over the English Wikipedia, and (3) taking the top passage that contains the answer. Based on our manual verification on 100 random samples, the precision of the silver evidence passages is 70%. We provide silver evidence passages primarily for supporting training of the model, and do not explicitly evaluate passage prediction; more discussion in Appendix A. Future work may use human annotations on top of our silver evidence passages in order to further improve the quality, or evaluate passage prediction. In order to evaluate the quality of claims and labels, three native English speakers were given 300 random samples from FAVIQ, and were asked to: (1) verify whether the claim is as natural as a human-written claim, with three possible ratings (perfect, minor issues but comprehensible, incom-prehensible), and (2) predict the label of the claim ( support or refute ). Validators were allowed to use search engines, and were encouraged to use the English Wikipedia as a primary source. Validators found 80.7% of the A set and 89.3% of the R set to be natural, and 0% to be incomprehensible. The rest have minor grammatical errors or typos, e.g., missing the. In most cases the errors actually come from the original NQ questions which were human-authored, indicating that these grammatical errors and typos occur in real life. Lastly, validators achieved an accuracy of 95.0% (92.7% of A and 97.3% of R) when evaluated against gold labels in the datathis indicates high-quality of the data and high human performance. This accuracy level is slightly higher than that of FEVER (91.2%). Data statistics for FAVIQ are listed in Table 1. It has 188k claims in total, with balanced support and refute labels. We present quantitative and qualitative analyses showing that claims on FAVIQ contain much less lexical bias than other crowdsourced datasets and include misinformation that is realistic and harder to identify. Comparison of size and claim length Table 2 compares statistics of a variety of fact verification datasets: SNOPES (Hanselowski et al., 2019), SCIFACT (Wadden et al., 2020), FEVER (Thorne et al., 2018a), FM2 (Eisenschlos et al., 2021), BOOLQ-FV (Thorne et al., 2021) and FAVIQ. FAVIQ is as large-scale as FEVER, while its distributions of claim length is much closer to claims authored by professional fact checkers (SNOPES and SCIFACT). FM2 is smaller scale, due to diffi-culty in scaling multi-player games used for data construction, and has claims that are slightly longer than professional claims, likely because they are intentionally written to be difficult. BOOLQ-FV is smaller, likely due to relative difficulties in collecting naturally-occurring yes/no questions. Lexical cues in claims We further analyze lexical cues in the claims on FEVER, FM2, BOOLQ-FV and FAVIQ by measuring local mutual information (LMI; Schuster et al. (2019); Eisenschlos et al. (2021)). LMI measures whether the given bigram correlates with a particular label. More specifically, LMI is defined as: LMI ( w, c ) = P ( w, c ) log P ( w, c ) P ( w )  P ( c ) , where w is a bigram, c is a label, and P (  ) are estimated by counting (Schuster et al., 2019). The distributions of the LMI scores for the top-100 bigrams are shown in Figure 3. The LMI scores of FAVIQ are significantly lower than those of FEVER, FM2, and BOOLQ-FV, indicating that FAVIQ contains significantly less lexical bias. Tables 3 shows the top six bigrams with the highest LMI scores for FEVER and FAVIQ. As highlighted, all of the top bigrams in refute claims of FEVER contain negative expressions, e.g., is only, incapable of, did not. In contrast, the top bigrams from FAVIQ do not include obvious negations and mostly overlap across different labels, strongly suggesting the task has fewer lexical cues. Although there are still top bigrams from FAVIQ causing bias (e.g., related to time, such as on October'), their LMI values are significantly lower compared those from other datasets. Qualitative analysis of the refute claims We also analyzed 30 randomly sampled refute claims from FAVIQ and FEVER respectively. We categorized the cause of misinformation as detailed in Appendix B, and show three most common categories for each dataset as a summary in Table 4. On FAVIQ, 60% of the claims involve entities, events or properties that are semantically close, but still distinct. For example, they are specified with conjunctions (e.g., was foreign minister and signed the treaty of versailles from germany), or share key attributes (e.g., films with the same ti-tle). This means that relying on lexical overlap or partially understanding the evidence text would lead to incorrect predictions; one must read the full evidence text to realize that the claim is false. Furthermore, 16.7% involve events, e.g., from fil-ing for bankruptcy for the first time to completely ceasing operations (Table 4). This requires full understanding of the underlying event and tracking of state changes (Das et al., 2019; Amini et al., 2020). (2021); many of claims contain explicit negations (30%) and antonyms (13%), with misinformation that is less likely to occur in the real world (20%). 7 4 Experiments We first evaluate state-of-the-art fact verification models on FAVIQ in order to establish baseline performance levels (Section 4.1). We then conduct experiments on professional fact-checking datasets to measure the improvements from training on FAVIQ (Section 4.2). We experiment with two settings: a zero-shot setup where models are trained on FEVER, and a standard setup where models are trained on FAVIQ. For FEVER, we use the KILT (Petroni et al., 2021) version following prior work; we randomly split the official validation set into equally sized validation and test sets, as the official test set is hidden. 7 For instance, consider the claim Mutiny on the Bounty is Dutch in Table 4. There is no Dutch producer, director, writer, actors, or actress in the filmwe were not able to find a potential reason that one would believe that the film is Dutch. All models are based on BART (Lewis et al., 2020), a pretrained sequence-to-sequence model which we train to generate either support or refute . We describe three different variants which differ in their input, along with their accuracy on FEVER by our own experiments. Claim only BART takes a claim as the only input. Although this is a trivial baseline, it achieves an accuracy of 79% on FEVER. TF-IDF + BART takes a concatenation of a claim and k passages retrieved by TF-IDF from Chen et al. (2017). It achieves 87% on FEVER. We choose TF-IDF over other sparse retrieval methods like BM25 (Robertson and Zaragoza, 2009) because Petroni et al. (2021) report that TF-IDF outperforms BM25 on FEVER. DPR + BART takes a concatenation of a claim and k passages retrieved by DPR (Karpukhin et al., 2020), a dual encoder based model. It is the state-of-the-art on FEVER based on Petroni et al. (2021) and Maillard et al. (2021), achieving an accuracy of 90%. Implementation details We use the English Wikipedia from 08/01/2019 following KILT (Petroni et al., 2021). We take the plain text and lists provided by KILT and create a collection of passages where each passage has up to 100 tokens. This results in 26M passages. We set the number of input passages k to 3, following previous work (Petroni et al., 2021; Maillard et al., 2021). Baselines on FAVIQ are jointly trained on the A set and the R set. Training DPR requires a positive and a negative passagea passage that supports and does not support the verdict, respectively. We use the silver evidence passage associated with FAVIQ as a positive, and the top TF-IDF passage that is not the silver evidence passages as a negative. More training details are in Appendix C. Experiments are reproducible from https://github.com/ faviq/faviq/tree/main/codes . Table 5 reports results on FAVIQ. The overall accuracy of the baselines is low, despite their high performance on FEVER. The zero-shot performance is barely better than random guessing, indicating that the model trained on FEVER is not able to generalize to our more challenging data. When the baselines are trained on FAVIQ, the best model 5159 Model Dev Test A R A R Training on FEVER (zero-shot) Clain only BART 51.6 51.0 51.9 51.1 TF-IDF + BART 55.8 58.5 54.4 57.2 DPR + BART 56.0 62.3 55.7 61.2 Training on FAVIQ Claim only BART 51.0 59.5 51.3 59.4 TF-IDF + BART 65.1 74.2 63.0 71.2 DPR + BART 66.9 76.8 64.9 74.6 Table 5: Fact verification accuracy on FAVIQ. achieves an accuracy of 65% on the A set, indicating that existing state-of-the-art models do not solve our benchmark. 8 Impact of retrieval The performance of the claim only baseline that does not use retrieval is almost random on FAVIQ, while achieving nearly 80% accuracy on FEVER. This result suggests significantly less bias in the claims, and the relative importance of using background knowledge to solve the task. When retrieval is used, DPR outperforms TF-IDF, consistent with the finding from Petroni et al. (2021). A set vs. R set The performance of the models on the R set is consistently higher than that on the A set by a large margin, implying that claims based on ambiguity arisen from real users are more challenging to verify than claims generated from regular question-answer pairs. This indicates clearer contrast to prior work that converts regular QA data to declarative sentences (Demszky et al., 2018; Pan et al., 2021). Error Analysis We randomly sample 50 error cases from DPR + BART on the A set of FAVIQ and categorize them, as shown in Table 6. Retrieval error is the most frequent type of errors. DPR typically retrieves a passage with the correct topic (e.g., about Lie to Me) but that is missing more specific information (e.g., the end date). We think the claim having less lexical overlap with the evidence text leads to low recall@ k of the retrieval model ( k = 3 ). 8 We additionally show and discuss the model trained on FAVIQ and tested on FEVER in Appendix D. They achieve non-trivial performance (67%) although being worse than FEVER-trained models that exploit bias in the data. 28% of error cases involve events . In particular, 14% involve procedural events, and 6% involve distinct events that share similar properties but differ in location or time frame. In 18% of error cases, retrieved evidence is valid but not notably explicit , which is naturally the case for the claims occurring in real life. FAVIQ has this property likely because it is derived from questions that are gathered independently from the evidence text, unlike prior work (Thorne et al., 2018a; Schuster et al., 2021; Eisenschlos et al., 2021) with claims written given the evidence text. 16% of the failure cases require multi-hop inference over the evidence. Claims in this category usually involve procedural events or compositions (e.g. is Seth Curry's brother and played for Davidson in college). This indicates that we can construct a substantial portion of claims requiring multi-hop inference without having to make data that artificially encourages such reasoning (Yang et al., 2018; Jiang et al., 2020). Finally, 10% of the errors were made due to a subtle mismatch in properties , e.g., in the example in Figure 6, the model makes a decision based on required minimum number rather than exact number of a particular brand. SNOPES (Hanselowski et al., 2019) consists of 6,422 claims, authored and labeled by professional fact-checkers, gathered from the Snopes website. 9 We use the official data split. SCIFACT (Wadden et al., 2020) consists of 1,109 claims based on scientific papers, annotated by domain experts. As the official test set is hidden, we use the official validation set as the test set, and separate the subset of the training data as the validation set to be an equal size as the test set. For both datasets, we merge not enough info ( NEI ) to refute , following prior work that converts the 3-way classification to the 2-way classification (Wang et al., 2019; Sathe et al., 2020; Petroni et al., 2021). the evidence text and is trained to generate either support or refute . For SNOPES , the evidence text is given in the original data. For SCIFACT, the evidence text is retrieved by TF-IDF over the corpus of abstracts from scientific papers, provided in the original data. We use TF-IDF over DPR because we found DPR works poorly when the training data is very small. We consider two settings. In the first setting, we assume the target training data is unavailable and compare the model trained on FEVER and FAVIQ in a zero-shot setup. In the second setting, we allow training on the target data and compare the model trained on the target data only and the model with the transfer learningpretrained on either FEVER or FAVIQ and finetuned on the target data. To explore models pretrained on NEI labels, we add a baseline that is trained on a union of the KILT version of FEVER and NEI data from the original FEVER from Thorne et al. (2018a). For FAVIQ, we also conduct an ablation that includes the R set only or the A set only. Implementation details When using TF-IDF for SCIFACT, we use a sentence as a retrieval unit, and retrieve the top 10 sentences, which average length approximates that of 3 passages from Wikipedia. When using the model trained on either FEVER or FAVIQ, we use DPR + BART by default, which gives the best result in Section 4.1. As an exception, we use TF-IDF + BART on SCIFACT for a more direct comparison with the model trained on the target data only that uses TF-IDF. When the models trained on FEVER or FAVIQ are used for professional fact checking, we find models are poorly calibrated, likely due to a domain shift, as also observed by Kamath et al. (2020) and Desai and Durrett (2020). We therefore use a simplified version of Platt scaling, a post-hoc calibration method (Platt et al., 1999; Guo et al., 2017; Zhao et al., 2021). Given normalized probabilities of support and refute , denoted as p s and p r , modified probabilities p (cid:48) s and p (cid:48) r are obtained via: (cid:20) p (cid:48) s p (cid:48) r (cid:21) = Softmax (cid:18)(cid:20) p s +  p r (cid:21)(cid:19) , where  1 <  < 1 is a hyperparameter tuned on the validation set. Impact of transfer learning We find that transfer learning is effectivepretraining on large, crowdsourced datasets (either FEVER or FAVIQ) and finetuning on the target datasets always helps. Improvements are especially significant on SCIFACT, likely because its data size is smaller. Using the target data is still importantmodels finetuned on the target data outperform zero-shot models by up to 20%. This indicates that crowdsourced data cannot completely replace professional fact checking data, but transfer learning from crowdsourced data leads to significantly better professional fact checking performance. FAVIQ vs. FEVER Models that are trained on FAVIQ consistently outperform models trained on FEVER, both with and without the target data, by up to 4.8% absolute. This demonstrates that FAVIQ is a more effective resource than FEVER for professional fact-checking. The model on FEVER is more competitive when NEI data is included, by up to 3% absolute. While the models on FAVIQ outperform models on FEVER even without NEI data, future work can possibly create NEI data in FAVIQ for further improvement. Impact of the A set in FAVIQ The performance of the models that use FAVIQ substantially degrades when the A set is excluded. Moreover, models trained on the A set (without R set) perform moderately well despite its small scale, e.g., on SNOPES , achieving the second best performance following the model trained on the full FAVIQ. This demonstrates the importance of the A set created based on ambiguity in questions. SNOPES benefits more from the A set than the R set, while SCIFACT benefits more from the R set than the A set. This is likely because SCIFACT is much smaller-scale (1k claims) and thus benefits more from the larger data like the R set. This suggests that having both the R set and the A set is important for performance. information-seeking questions. We incorporate facts that real users were unaware of when posing the question, leading to false claims that are more realistic and challenging to identify without fully understanding the context. Our extensive analysis shows that our data contains significantly less lexical bias than previous fact checking datasets, and include refute claims that are challenging and realistic. Our experiments showed that the state-of-the-art models are far from solving FAVIQ, and models trained on FAVIQ lead to improvements in professional fact checking. Altogether, we believe FAVIQ will serve as a challenging benchmark as well as support future progress in professional fact-checking. We suggest future work to improve the FAVIQ model with respect to our analysis of the model prediction in Section 4.1.2, such as improving retrieval, modeling multi-hop inference, and better distinctions between entities, events and properties. Moreover, future work may investigate using other aspects of information-seeking questions that reflect facts that users are unaware of or easily confused with. For example, one can incorporate false presuppositions in questions that arise when users have limited background knowledge (Kim et al., 2021). As another example, one can explore generating NEI claims by leveraging unanswerable information-seeking questions. Furthermore, FAVIQ can potentially be a challenging benchmark for the claim correction, a task recently studied by Thorne and Vlachos (2021) that requires a model to correct the refute claims. We thank Dave Wadden, James Thorn and Jinhyuk Lee for discussion and feedback on the paper. We thank James Lee, Skyler Hallinan and Sourojit Ghosh for their help in data validation. This work was supported by NSF IIS-2044660, ONR N00014-18-1-2826, an Allen Distinguished Investigator Award, a Sloan Fellowship, National Research Foundation of Korea (NRF-2020R1A2C3010638) and the Ministry of Science and ICT, Korea, under the ICT Creative Consilience program (IITP-2022-2020-0-01819).
abstain abstain method abstain abstain objective result abstain abstain abstain other abstain abstain result abstain abstain objective abstain abstain method abstain abstain result result result abstain objective objective abstain abstain objective objective other other other other other other other other method abstain abstain other other method other method abstain method abstain abstain method abstain abstain abstain abstain method abstain method other method abstain method abstain method abstain method abstain method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain method other abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain abstain abstain method abstain method abstain method method abstain method abstain method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method result objective method result abstain abstain abstain abstain other other other