Computing precise evidences , namely minimal sets of sentences that support or refute a given claim, rather than larger evidences is crucial in fact verification (FV), since larger evidences may contain conflicting pieces some of which support the claim while the other refute, thereby misleading FV. Despite being important, precise evidences are rarely studied by existing methods for FV. It is challenging to find precise evidences due to a large search space with lots of local optimums. Inspired by the strong exploration ability of the deep Q-learning network (DQN), we propose a DQN-based approach to retrieval of precise evidences. In addition, to tackle the label bias on Q-values computed by DQN, we design a postprocessing strategy which seeks best thresholds for determining the true labels of computed evidences. Experimental results confirm the effectiveness of DQN in computing precise evidences and demonstrate improvements in achieving accurate claim verification. 1 1 Introduction With the growing false information, such as fake news, political deception and online rumors, automatic fact-checking systems have emerged to automatically identify and filter this information. Fact verification (FV) is a special fact-checking task that aims to retrieve related evidences from a text corpus to verify a textual claim. Taking Figure 1 as example, an existing method for FV first retrieves related documents from the given corpus at stage 1 (namely the document retrieval stage), then finds key sentences from the documents at stage 2 (namely the sentence selection stage), and finally treats the set of key sentences as an evidence to verify the claim at stage  Corresponding author 1 Source code and data are available at https:// github.com/sysulic/DQN-FV . 3 (namely the claim verification stage). As can be seen in this example, it is desirable to retrieve an evidence consisting of the first two sentences only, since it does not contain unnecessary sentences to determine the truthfulness of the claim and can alleviate human efforts to further validate the evidence. More importantly, an evidence containing unnecessary sentences may involve conflicting pieces some of which support the claim while the other refute the claim. Thus, it is crucial to compute minimal sets of sentences that can determine the truthfulness of the claim. In this paper, we refer to a minimal set of sentences that supports or refutes a given claim as a precise evidence . Existing methods for FV do not target the retrieval of precise evidences. Most existing studies (Thorne et al., 2018b; Nie et al., 2019; Zhou et al., 2019; Liu et al., 2020; Zhong et al., 2020; Ye et al., 2020; Subramanian and Lee, 2020; Wang et al., 2020) formulate FV as a three-stage pipeline task as illustrated in Figure 1. This way makes the retrieval of precise evidences extremely diffi-cult since the sentence selection stage is required to select a precise set of relevant sentences rather than a fixed number of sentences as in existing methods. To the best of our knowledge, TwoWingOS (Yin and Roth, 2018) is the only method by now which does not follow the three-stage pipeline. Instead, it exploits a supervised training scheme to train the last two stages jointly and is able to compute precise evidences. However, it exhibits a significantly worse performance than other state-of-the-art methods for FV, especially in terms of the recall of evidences. Therefore, there is still a need for designing new methods to compute precise evidences. These methods are expected to achieve better performance than TwoWingOS. It is challenging to compute precise evidences. On one hand, the search space for precise evidences is very large. For example, in the benchmark Fact Extraction and VERification (FEVER) dataset (Thorne et al., 2018b) the average number of sentences for each claim input to the sentence selection stage is 40 , and an output evidence has up to 5 sentences. Hence there are up to (cid:80) 5 i =1 C i 40 = 760 , 098 candidates in the search space. On the other hand, greedy search of precise evidences easily falls into a local optimum. As shown in our experiments (see Table 6), a greedy search method does not perform well. Inspired by the strong exploration ability of the Deep Q-learning Network (DQN) (Mnih et al., 2015), we develop a DQN-based approach to retrieval of precise evidences. In this approach, we first employ DQN to compute candidate pairs of precise evidences and their labels, and then use a post-processing strategy to refine candidate pairs. We notice that Q-values computed by DQN has label bias due to two reasons. On one hand, the label NOT ENOUGH INFO does not locate at the same concept level as SUPPORTS or REFUTES. On the other hand, there is not a fixed range for Q-values, making Q-values hard to accurately estimate. Thus, a post-processing strategy is needed to tackle the label bias on Q-values. We develop such a strategy to seek best thresholds in determining the true labels of computed evidences. Our experimental results on FEVER (Thorne et al., 2018b) confirm that our DQN-based approach is effective in finding precise evidences. More importantly, the approach is shown to outperform state-of-the-art methods for FV. The FEVER 1.0 shared task (Thorne et al., 2018b) aims to develop an automatic fact verification system to determine the truthfulness of a textual claim by extracting related evidences from Wikipedia. Thorne et al. (2018a) has formalized this task, released a large-scale benchmark dataset FEVER (Thorne et al., 2018b), and designed the three-stage pipeline framework for FV, which consists of the document retrieval stage, the sentence selection stage and the claim verification stage. Most existing methods follow this framework and mainly focus on the last stage (Liu et al., 2020). For the document retrieval stage, most methods reuse the document retrieval component of top-performing systems (Hanselowski et al., 2018; Yoneda et al., 2018; Nie et al., 2019). For the sentence selection stage, there are three approaches commonly used, including keyword matching, supervised classification, and sentence similarity scoring (Thorne et al., 2018b). For the claim verification stage, most recent studies formulate this task as a graph reasoning task (Zhou et al., 2019; Liu et al., 2020; Ye et al., 2020; Zhong et al., 2020; Subramanian and Lee, 2020; Wang et al., 2020). Different from most existing methods that focus on claim verification, Yin and Roth (2018) proposed a supervised training method named TwoWingOS to jointly conduct sentence selection and claim verification. Nowadays pre-trained language models like BERT (Devlin et al., 2019) have been widely used in claim verification (Li et al., 2019; Zhou et al., 2019; Soleimani et al., 2020). Following this way we employed RoBERTa (Liu et al., 2019), an enhanced version of BERT, as the sentence encoder in our DQN-based approach in experiments. Reinforcement learning (RL) is about an agent interacting with the environment, objective to maximize the cumulative rewards of a sequence of states and actions by adjusting its policies. Q-Learning (Mnih et al., 2015) is a popular reinforcement learning technique. It aims to approximate the optimal value function Q  ( o, a ) to measure the expected long-term rewards for a given pair of state o and action a . Deep Q-learning Network (DQN) (Mnih et al., 2015) is a combination of deep learning and Q-Learning. It typically uses the following Equation (1) derived from the Bellman equation (Cao and ZhiMin, 2019) to approximate the optimal Q-value function: Q ( o ( t ) , a ( t ) ) = E o ( t +1) [ r ( t ) +  max a (cid:48) Q ( o ( t +1) , a (cid:48) )] , (1) where o ( t ) , a ( t ) , r ( t ) respectively denote the state, action and reward at step t , and   [0 , 1] is a discounted factor for future rewards. Given a set of candidate sentences S = { s 1 , s 2 , . . . } , a claim c , a set of precise evidences E  2 S , and a true label y  Y = { T,F,N } that determines whether every precise evidence supports or refutes the claim, where T / F / N denotes SUP-PORTS/REFUTES/NOT ENOUGH INFO, we aim to train a model to predict a precise evidence; more precisely, to train a model for retrieving an evidence  E  S and predicting a label  y  Y such that  y = y and  E = E for some E  E . This goal is different from the goal targeted by existing methods, which aim to retrieve an evidence  E  S and predict a label  y  Y such that  y = y and E   E for some E  E . We define the four ingredients of DQN namely states, actions, transitions and rewards as follows:  State . Action . An action a is a sentence in S . Transition . A transition at step t is a tuple ( o ( t ) , a ( t ) , o ( t +1) ) , where o ( t ) = ( c,  E ( t ) ,  y ) , o ( t +1) = ( c,  E ( t +1) ,  y ) and  E ( t +1) =  E ( t )  { a ( t ) } . Reward . The reward r for a transition ( o ( t ) , a ( t ) , o ( t +1) ) is defined as r ( t ) =   1 ,  y = y  ( y = N  E  E : a ( t )  E )  1 ,  y (cid:54) = y |  E ( t +1) | = K 0 , otherwise (2) where the number K is a hyper-parameter, and | S | denotes the cardinality of a set S . The core of our proposed approach is the DQN-based model, illustrated in Figure 2. 3.2.1 Sentence Encoding Module We employ RoBERTa in this module to extract the final hidden state of (cid:104) s (cid:105) as the sentence representation, where (cid:104) s (cid:105) and (cid:104) /s (cid:105) mentioned in the following are the special classification tokens in RoBERTa. Specifically, following KGAT (Liu et al., 2020), we first concatenate the claim c , the document title l , and a sentence s ( resp. an action a ) as  (cid:104) s (cid:105) c (cid:104) /s (cid:105) l (cid:104) /s (cid:105) s (cid:104) /s (cid:105)  ( resp. (cid:104) s (cid:105) c (cid:104) /s (cid:105) l (cid:104) /s (cid:105) a (cid:104) /s (cid:105) ) and then feed it into RoBERTa to obtain the sentence representation h s  R d 0 ( resp. the action representation h a  R d 0 ), where d 0 is the dimension of the representation. We also feed the claim  (cid:104) s (cid:105) c (cid:104) /s (cid:105)  alone to obtain the claim representation h c  R d 0 . Context sub-module . It is obvious that the sentences in an evidence are always contextual dependent, so we apply two different networks BiLSTM (Nguyen et al., 2016) and Transformer (Vaswani et al., 2017) for comparison. These two different networks are widely used to encode contextual-aware information of sequential text in the NLP community. Formally, we either define [ h (cid:48)  E 0 , . . . , h (cid:48)  E |  E | 1 ] = BiLSTM ( h a , H  E ) (3) if the BiLSTM network is used, or define [ h (cid:48)  E 0 , . . . , h (cid:48)  E |  E | 1 ] = Transformer ( h a , H  E ) (4) if the Transformer is used, where H  E = [ h  E 0 , . . . , h  E |  E | 1 ] , h  E i  R d 0 is the i -th sentence representation in  E , h (cid:48)  E i  R d 1 is the corresponding context-aware sentence representation in  E , and d 1 is the dimension of the representation. Aggregation sub-module . This sub-module is used to fuse the sentence representations in evidences to obtain an aggregated evidence representation. We also apply two different networks in this sub-module: Transformer and attention. Unlike the Transformer with self-attention in the first submodule, the query in this sub-module is the claim and the key/value is the context-aware sentence representation from the first sub-module. For the Figure 2: The architecture of the DQN-based model. e = |  E | 1 (cid:88) i =0  i  h (cid:48) i (5)  i = exp( MLP ([ h c ; h (cid:48) i ])) |  E | 1 (cid:88) j =0 exp( MLP ([ h c ; h (cid:48) j ])) (6) where e  R d 1 is the aggregated evidence representation, MLP (  ) = Linear ( ReLU ( Linear (  ))) is a two-layer fully connected network using recti-fied linear unit as the activation function, and [; ] denotes the concatenation of two vectors. This module is used to obtain the Q-value vector for all labels, simply written as Q ( o, a ;  ) for  denoting the set of learnable parameters, which is formally defined as where MLP (  ) = Linear ( ReLU ( Linear (  ))) is similar to MLP (  ) used in Equation (6) except that different parameters in linear layers are used, W  R d 0  d 0 is a learnable matrix, and Q ( o, a ;  )  R d 2 for d 2 the number of different labels. Given a transition ( o ( t ) , a ( t ) , o ( t +1) ) and its reward r ( t ) , we use the Double Deep Q-learning Network (DDQN) (Mnih et al., 2015) technique to train our et al., 2015). This error  is formally defined as  = Q  y ( o ( t ) , a ( t ) ;  )  v ( o ( t +1) , r ( t ) ) (8) where v (  ) denotes the target value defined as v ( o, r ) = (cid:40) r, if |  E | = K r +   Q  y ( o, a  ;   ) otherwise (9) for a  = arg max a  S \  EQ  y ( o, a ;  ) . In the above equation,  Q (  ;   ) is the target network in DDQN, Q  y denotes the Q-value of  y for  y the predicted label in o ,  E is the predicted evidence in o , and   [0 , 1] is a hyper-parameter representing the discount factor. where B is a batch of transition-reward pairs. Algorithm 1 shows how to train the DQN-based model. First, we initialize three replay memories, the DQN-based model, and the target network in Line 1-3. Then, in Line 9-17, we obtain the training Algorithm 1: Model training for DQN, where the memory capacity M , the maximum evidence size K , the maximum number of epochs T and the reset interval C are hyper-parameters. transition-reward pairs by letting the DQN-based model interact with the environment in an (cid:15) -greedy exploration-exploitation way (Mnih et al., 2015). Finally, in Line 19, we sample a mini-batch of transition-reward pairs to update the DQN-based model, while in Line 20, for every C steps we reset the target network to the DQN-based model. Algorithm 2 shows how to retrieve a pair (candi-date list, score list) for each label, where the can-Algorithm didate list stores progressively enlarged sentence sets, where each sentence set is a candidate of the predicted evidence, and the score list stores the strengths that the corresponding candidates support the label. We enlarge the two-list pair for each label through a greedy-search way (Line 3-10). Specifi-cally, for each label, we first select the action with the largest Q-value (Line 5), then update the state by adding the chosen action into its predicted evidence (Line 7), and finally add the evidence and score into the corresponding list (Line 8). Algorithm 3 shows how to compute the target evidence-label pair from the (candidate list, score list) pairs obtained by Algorithm 2, where the thresholds are determined by Algorithm 4. In this algorithm, we first use the condition given by Algorithm 4 to predict N (Line 3), and then refine the prediction of T (Line 6) and F (Line 9) in turn. In Line 2, we focus on the evidences with the highest score for T and F , while we ignore the evidence for N , due to the following reasons: (1) there are no supporting sentences in the evidence for N ; (2) we follow a strategy commonly used in existing methods for FV, i.e. , focusing only on the evidence for T and F . Algorithm 4 shows how to search for the best thresholds (  T ,  F ,  N ) to maximize the Label Accuracy (LA) over the development set. We first call Algorithm 2 to construct a set of tuples ( q T , q F , q N , y ) from the development set, each of which corresponds to a development instance, where q T , q F and q N are respectively the output score lists for the three labels T, F and N , and y is the corresponding true label (Line 1). We then go through the following two stages. The first stage (Line 3-11) finds a threshold  N that can maximize LA for label N , where maximizing LA is amount to maximizing the difference between the number of correctly and incorrectly predicted instances. The second stage (Line 12-28) finds the thresholds  T and  F that can maximize LA for label T and F , respectively, where those instances that satisfy the conditions for N are neglected (Line 13). Our experiments are conducted on the large-scale benchmark dataset FEVER (Thorne et al., 2018a), which consists of 185,455 annotated claims with a set of 5,416,537 Wikipedia documents from the June 2017 Wikipedia dump. All claims are labeled as SUPPORTS, REFUTES, or NOT ENOUGH INFO. What's more, each claim for SUPPORTS and REFUTES is accompanied by some evidences extracted from Wikipedia documents. The dataset partition is kept the same with Thorne et al. (2018b) as shown in Table 1. 4.1.2 Evaluation Metrics The task has five evaluation metrics: 1) FEVER, the primary scoring metric that measures the accuracy of claim verification with a requirement that the predicted evidences fully covers the ground-true evidences for SUPPORTS and REFUTES claims; 2) Label Accuracy (LA), the accuracy of claim verification without considering the validity of the Method  T  F  NT-T -1.23361155390739 -1.26671668887138 0.0153777748346328 T-A -0.0631487071514129 0.0747150778770446 -1.48811344802379 BiLSTM-T 0.184351719915866 -0.64785711467266 -0.465365642681717 BiLSTM-A -0.0904324240982532 -0.795884847640991 -0.403448916971683 Table 2: The thresholds determined by Algorithm predicted evidences; 3) Precision (Pre), the macro-precision of the evidences for SUPPORTS and REFUTES claims; 4) Recall, the macro-recall of the evidences for SUPPORTS and REFUTES claims; 5) F1, the F 1 -score of the evidences for SUPPORTS and REFUTES claims. We choose F1 as our main metric because it can directly show the performance of methods on retrieval of precise evidences. Document retrieval . The document retrieval stage is kept the same as previous work (Hanselowski et al., 2018; Zhou et al., 2019; Liu et al., 2020; Ye et al., 2020). Given a claim, the method first utilizes the constituency parser from AllenNLP (Gardner et al., 2018) to extract potential entities from the claim. Then it uses the entities as search queries to find the relevant documents via the online Me-diaWiki API 2 . The convinced articles are reserved (Hanselowski et al., 2018). Sentence selection and claim verification . We implement our DQN-based model with PyTorch and train it with the AdamW (Loshchilov and Hut-ter, 2019) optimizer while keeping the sentence encoding module frozen and inheriting the RoBERTa implementation from Wolf et al. (2020) 3 . Specifically, the learning rate is 5e-6, the batch size is 128, the training epochs is 30, the iteration steps (or largest evidence size, i.e. , K ) is 5, the discount factor  is 0.95, and the layer number of the context sub-module is 3. Prioritized experience replay memory (Schaul et al., 2016) with a capacity of 10,000 is used to store transitions. The target network is reset when DQN is updated every 10 times. The probability of (cid:15) -greedy policy starts at 0.9 and decays exponentially towards 0.05, and the rate of the decay is 1 2000 . Table 2 shows the thresholds 2 https://www.mediawiki.org/wiki/API: Main_page 3 https://github.com/huggingface/ pytorch-transformers  T ,  F and  N computed by Algorithm We compare our method with the following baselines, including six methods that focus on claim verification and one joint method TwoWingOS (Yin and Roth, 2018). The six methods include: (1) GEAR (Zhou et al., 2019) uses two kinds of attentions to conduct reasoning and aggregation in a graph model; (2) KGAT (Liu et al., 2020) employes the Kernel Graph Attention Network to capture fine-grained information over evidences for more accurate claim verification; (3) DREAM (Zhong et al., 2020) introduces semantic structures for evidences obtained by semantic role labeling in claim verification; (4) CorefBERT (Ye et al., 2020) extends KGAT and can explicitly model co-reference relationship in context; (5) HESM (Subramanian and Lee, 2020) is a framework that can encode and attend the claim and evidence sets at different levels of hierarchy; (6) DGAT (Wang et al., 2020) is a double graph attention network that performs well in multi-domain datasets. The join method TwoWingOS (Yin and Roth, 2018) exploits a two-wing optimization strategy that optimizes sentence selection and claim verification in a jointly supervised training scheme. As shown in Table 3, we implement four versions of the evidence encoding module and evaluate them on the DEV set and the blind TEST set. The FEVER metric of the top six methods is calculated with the imprecise evidences, so we introduce the FEVER@5 metric for a fair comparison. We analyze our method from the following four aspects. Comparison with the state-of-the-art methods . Results in Table 3 show that all versions (except BiLSTM-A) with post-processing significantly outperform the state-of-the-art methods on FEVER, Pre, and F1, especially for T-A on F1, which shows the superiority of our method in retrival of precise evidences. However, none of the four versions of our method can achieve the best result on FEVER@5, LA, and Recall. The reason for low recall is that the number of sentences in precise evidences is less than that in imprecise evidences, which means other methods have a higher probability to recall the ground-true evidences than ours. Besides, the relatively low LA is caused by the Method DEV TEST FEVER@5 FEVER LA Pre Recall F1 FEVER@5 FEVER LA Pre Recall F1 GEAR 70.69 -74.84 24.08 86.72 37.69 67.10 -71.60 -36.87 KGAT 76.11 -78.29 27.79 94.37 42.34 70.38 -74.07 25.21 87.47 39.14 DREAM --26.60 87.33 40.79 70.60 -76.85 25.63 85.57 39.45 CorefBERT ---71.80 --39.14 HESM 73.44 -75.77 --71.48 -74.64 -52.78 DGAT ---66.91 -71.79 -TwoWingOS -56.16 78.90 47.73 53.81 50.59 -54.33 75.99 44.68 49.91 47.15 Ours T-T 72.83 71.55 78.18 50.42 81.82 62.39 70.16 68.91 75.74 48.76 79.91 60.56 (w./o.) 72.90 70.00 74.87 70.43 68.23 73.13 T-A 73.32 72.79 78.35 54.75 79.92 64.98 70.81 70.28 76.14 52.24 77.93 62.55 (w./o.) 73.29 72.60 78.12 70.82 70.18 76.00 BiLSTM-T 73.15 63.77 73.91 48.06 71.06 57.34 70.54 61.51 70.20 45.97 69.43 55.32 (w./o.) 73.19 55.39 63.55 70.81 53.21 61.68 BiLSTM-A 72.99 70.88 77.79 35.50 76.54 48.50 70.11 68.21 75.53 33.76 74.50 46.46 (w./o.) 73.20 65.65 71.21 70.55 63.38 69.32 Table 3: Performance on DEV set and blind TEST set of FEVER (%). low Recall of precise evidences. To further clarify this point, we evaluate our method on a subset of the DEV set where the ground-true evidences are recalled successfully. Our method improves significantly the performance on this subset, as shown in Table 4, which justifies our point of view. FEVER is affected by the LA and Recall, thereby the low FEVER@5 is also due to the low recall of precise evidences. In addition, the results reported in Table 5 show that our method can significantly reduce the number of unnecessary sentences in a predicted evidence. Comparison between different versions . As shown in Table 3, T-T and T-A perform respectively better than BiLSTM-T and BiLSTM-A on almost all metrics except that T-T is slightly worse width FEVER@5 FEVER LA Pre Recall F1 1 60.73 54.91 72.69 52.76 58.57 55.51 (w./o.) 50.09 46.55 53.00 2 60.74 54.94 72.69 52.84 58.66 55.59 (w./o.) 50.09 46.53 53.00 3 60.70 54.96 72.69 52.84 58.67 55.60 (w./o.) 50.10 46.54 53.00 4 60.67 54.95 72.69 52.81 58.66 55.58 (w./o.) 50.09 46.54 53.00 5 60.68 54.95 72.69 52.84 58.68 55.61 (w./o.) 50.09 46.54 53.00 Table 6: The beam-search result of KGAT on the DEV set (%). than BiLSTM-A on FEVER@5, which suggests Transformer can encode better context-aware representations than BiLSTM in our context sub-module. Moreover, we find that T-A performs better than T-T on almost all metrics except Recall and that BiLSTM-A is worse than BiLSTM-T on Pre and F1. This contrary result shows that the performance of the aggregation sub-module is impacted by the context sub-module. Thus, the choice between Transformer and Attention should depend on the context sub-module. Overall, T-A achieves the best performance among all the four versions of our # label claim ground-true evidences predicted evidences GEAR KGAT Our method (T-A) 1 F Savages was exclusively a German film. Comparison on retrieval of precise evidences . TwoWingOS is a supervised-learning method that can also find precise evidences. Although it achieves slightly better performance on LA than ours, its F1 and other metrics are much worse, indicating that it performs worse than our method except for BiLSTM-A in retrieval of precise-evidences. We also enhance KGAT to conduct beam-search for finding precise evidences and report the results in Table 6. The F1 score of KGAT is always higher than TwoWingOS but is still lower than our method except for BiLSTM-A. Comparison between the methods with and without post-processing . It can be seen from Table 3 and Table 6 that, post-processing (namely threshold searching and final prediction from candidates) consistently improves FEVER and LA. Although with post-processing, our method (except T-A) achieves slightly lower scores on FEVER@5, KGAT still achieves significantly higher scores on FEVER@5 as on other metrics. These results show that post processing is very important in retrieval of precise evidences. In Table 7 we provide some cases to demonstrate the effectiveness of our method (T-A) in retrieving precise evidences. In case#1 and case#2, our method exactly finds ground-true evidences without introducing any unnecessary sentence, while GEAT and KGAT cannot. In case#3 and case#4, our method generates less unnessary sentences in prdicted evidents than GEAT and KGAT do. In this paper, we have proposed a novel DQN-based approach to finding precise evidences for fact verification. It provides a method to solve the precise-evidence problem by first employing a DQN to compute some candidates and then introducing a post-processing strategy to extract the target evidence and its label from the candidates. Experimental results show that the approach achieves state-of-the-art performance in terms of retrieval of precise evidences. Besides, to the best of our knowledge, it is the first attempt to employ DQN in the fact verification task. This work is supported by the Guangdong Province Science and Technology Plan projects (2017B010110011), the National Natural Science Foundation of China (No. 61876204, 61976232, and 51978675), the National Key R&D Program of China (No.2018YFC0830600), Guangdong Province Natural Science Foundation (No. 2018A030313086), All-China Federation of Returned Over-seas Chinese Research Project (No. 17BZQK216).
abstain abstain abstain objective method abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain objective method abstain abstain abstain objective result abstain other other other other other other other other method other other other other other method abstain method abstain abstain abstain other abstain other method method method abstain method abstain method abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method abstain other abstain abstain abstain method method abstain method abstain abstain method method abstain method method abstain method method abstain abstain abstain abstain abstain abstain other abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain other abstain result abstain abstain abstain result result abstain other abstain abstain result abstain result abstain other abstain result abstain abstain result abstain abstain method method method abstain abstain method abstain abstain result abstain objective abstain abstain objective other