Fact Verification requires fine-grained natural language inference capability that finds subtle clues to identify the syntactical and semantically correct but not well-supported claims. This paper presents Kernel Graph Attention Network (KGAT), which conducts more fine-grained fact verification with kernel-based attentions. Given a claim and a set of potential evidence sentences that form an evidence graph, KGAT introduces node kernels, which better measure the importance of the evidence node, and edge kernels, which conduct fine-grained evidence propagation in the graph, into Graph Attention Networks for more accurate fact verification. KGAT achieves a 70.38% FEVER score and significantly outperforms existing fact verification models on FEVER, a large-scale benchmark for fact verification. Our analyses illustrate that, compared to dot-product attentions, the kernel-based attention concentrates more on relevant evidence sentences and meaningful clues in the evidence graph, which is the main source of KGAT's effectiveness. All source codes of this work are available at https://github. com/thunlp/KernelGAT . Online contents with false information, such as fake news, political deception, and online rumors, have been growing significantly and spread widely over the past several years. How to automatically fact check the integrity of textual contents, to prevent the spread of fake news, and to avoid the undesired social influences of maliciously fabricated statements, is urgently needed for our society. Recent research formulates this problem as the fact verification task, which targets to automatically verify the integrity of statements using trustworthy corpora, e.g., Wikipedia (Thorne et al., 2018a). For example, as shown in Figure 1, a system could first Al Jardine is an American rhythm guitarist Claim Verification SUPPORTS REFUTES NOT ENOUGH INFO Evidence Reasoning Alan Charles Jardine (born September 3, 1942) is an American musician , singer and songwriter who cofounded the Beach Boys. retrieve related evidence sentences from the background corpus, conduct joint reasoning over these sentences, and aggregate the signals to verify the claim integrity (Nie et al., 2019a; Zhou et al., 2019; Yoneda et al., 2018; Hanselowski et al., 2018). There are two challenges for evidence reasoning and aggregation in fact verification. One is that no ground truth evidence is given; the evidence sentences are retrieved from background corpora, which inevitably contain noise. The other is that the false claims are often deliberately fabricated; they may be semantically correct but are not supported. This makes fact verification a rather challenging task, as it requires the fine-grained reasoning ability to distinguish the subtle differences between truth and false statements (Zhou et al., 2019). This paper presents a new neural structural reasoning model, Kernel Graph Attention Network (KGAT), that provides more fine-grained evidence selection and reasoning capability for fact verification using neural matching kernels (Xiong et al., 2017; Dai et al., 2018). Given retrieved evidence pieces, KGAT first constructs an evidence graph, using claim and evidence as graph nodes and fully-connected edges. It then utilizes two sets of kernels, one on the edges, which selectively summarize clues for a more fine-grained node representation and propagate clues among neighbor nodes through a multi-layer graph attention; and the other on the nodes, which performs more accurate evidence selection by better matching evidence with the claim. These signals are combined by KGAT, to jointly learn and reason on the evidence graph for more accurate fact verification. In our experiments on FEVER (Thorne et al., 2018a), a large-scale fact verification benchmark, KGAT achieves a 70.38% FEVER score, significantly outperforming previous BERT and Graph Neural Network (GNN) based approaches (Zhou et al., 2019). Our experiments demonstrate KGAT's strong effectiveness especially on facts that require multiple evidence reasoning: our kernel-based attentions provide more sparse and focused attention patterns, which are the main source of KGAT's effectiveness. The FEVER shared task (Thorne et al., 2018a) aims to develop automatic fact verification systems to check the veracity of human-generated claims by extracting evidence from Wikipedia. The recently launched FEVER shared task 1.0 is hosted as a competition on Codalab 1 with a blind test set and has drawn lots of attention from NLP community. Existing fact verification models usually employ FEVER's official baseline (Thorne et al., 2018a) with a three-step pipeline system (Chen et al., 2017a): document retrieval, sentence retrieval and claim verification. Many of them mainly focus on the claim verification step. Nie et al. (2019a) concatenates all evidence together to verify the claim. One can also conduct reasoning for each claim evidence pair and aggregate them to the claim label (Luken et al., 2018; Yoneda et al., 2018; Hanselowski et al., 2018). TwoWingOS (Yin and Roth, 2018) further incorporates evidence identifi-cation to improve claim verification. GEAR (Zhou et al., 2019) formulates claim verification as a graph reasoning task and provides two kinds of attentions. It conducts reasoning and aggregation over claim evidence pairs with 1 https://competitions.codalab.org/ competitions/18814 a graph model (Velickovic et al., 2017; Scarselli et al., 2008; Kipf and Welling, 2017). Zhong et al. (2019) further employs XLNet (Yang et al., 2019) and establishes a semantic-level graph for reasoning for a better performance. These graph based models establish node interactions for joint reasoning over several evidence pieces. Many fact verification systems leverage Natural Language Inference (NLI) techniques (Chen et al., 2017b; Ghaeini et al., 2018; Parikh et al., 2016; Radford et al., 2018; Peters et al., 2018; Li et al., 2019) to verify the claim. The NLI task aims to classify the relationship between a pair of premise and hypothesis as either entailment, contradiction or neutral, similar to the FEVER task, though the later requires systems to find the evidence pieces themselves and there are often multiple evidence pieces. One of the most widely used NLI models in FEVER is Enhanced Sequential Inference Model (ESIM) (Chen et al., 2017b), which employs some forms of hard or soft alignment to associate the relevant sub-components between premise and hypothesis. BERT, the pre-trained deep bidirectional Transformer, has also been used for better text representation in FEVER and achieved better performance (Devlin et al., 2019; Li et al., 2019; Zhou et al., 2019; Soleimani et al., 2019). The recent development of neural information retrieval models, especially the interaction based ones, have shown promising effectiveness in extracting soft match patterns from query-document interactions (Hu et al., 2014; Pang et al., 2016; Guo et al., 2016; Xiong et al., 2017; Dai et al., 2018). One of the effective ways to model text matches is to leverage matching kernels (Xiong et al., 2017; Dai et al., 2018), which summarize word or phrase interactions in the learned embedding space between query and documents. The kernel extracts matching patterns which provide a variety of relevance match signals and shows strong performance in various ad-hoc retrieval dataset (Dai and Callan, 2019). Recent research also has shown kernels can be integrated with contextualized representations, i.e., BERT, to better model the relevance between query and documents (MacAvaney et al., 2019). This section describes our Kernel Graph Attention Network (KGAT) and its application in Fact Verification. Following previous research, KGAT first constructs an evidence graph using retrieved evidence sentences D = { e 1 , . . . , e p , . . . , e l } for claim c , and then uses the evidence graph to predict the claim label y (Sec. 3.1 and 3.2). As shown in Figure 2, the reasoning model includes two main components: Evidence Propagation with Edge Kernels (Sec. 3.3) and Evidence Selection with Node Kernels (Sec. 3.4). Similar to previous research (Zhou et al., 2019), KGAT constructs the evidence graph G by using each claim-evidence pair as a node and connects all node pairs with edges, making it a fully-connected evidence graph with l nodes: N = { n 1 , . . . , n p , . . . , n l } . KGAT unifies both multiple and single evidence reasoning scenarios and produces a probability P ( y | c, D ) to predict claim label y . Different from previous work (Zhou et al., 2019), we follow the standard graph label prediction setting in graph neural network (Velickovic et al., 2017) and split the prediction into two components: 1) the label prediction in each node conditioned on the whole graph P ( y | n p , G ) ; 2) the evidence selection probability P ( n p | G ) : P ( y | c, D ) = l (cid:88) p =1 P ( y | c, e p , D ) P ( e p | c, D ) , (1) or in the graph notation: P ( y | G ) = l (cid:88) p =1 P ( y | n p , G ) P ( n p | G ) . The joint reasoning probability P ( y | n p , G ) calculates node label prediction with multiple evidence. The readout module (Knyazev et al., 2019) calculates the probability P ( n p | G ) and attentively combines per-node signals for prediction. The rest of this section describes the initialization of node representations ( n p ) in Sec. 3.2, the calculation of per-node predictions P ( y | n p , G ) with Edge Kernels (Sec. 3.3), and the readout module P ( n p | G ) with Node Kernels (Sec. 3.4). The node representations are initialized by feeding the concatenated sequence of claim, document (Wiki) title, and evidence sentence, to pre-trained BERT model (Devlin et al., 2019). Specifically, in the node n p , the claim and evidence correspond to m tokens (with [SEP]) and n tokens (with Joint Evidence Reasoning MLP MLP Claim Label Node Kernel " #(%|' 1 , *) #(%|' , , *) #(%|' , *) Evidence Reasoning Evidence Selection Edge Kernel ' 1 ' -' , #(%|G) / 0 #(' |*) 1(' 0 ) 2 3 2 4 2 0 / 4 #(' , |*) 1(' 4 ) / 3 #(' 1 |*) 1(' 3 ) Figure 2: KGAT Architecture. Wikipedia title and [SEP]) . Using the BERT encoder, we get the token hidden states H p with the given node n p : H p = BERT ( n p ) . The rest of the sequences H p 1: m + n are also used to represent the claim and evidence tokens: H p 1: m for the claim tokens and H pm +1: m + n for the evidence tokens. The evidence propagation and per-node label prediction in KGAT are conducted by Edge Kernels, which attentively propagate information among nodes in the graph G along the edges with the kernel attention mechanism. Specifically, KGAT calculates the node n p 's representation v p with the kernel attention mechanism, and uses it to produce the per-node claim prediction y : v p = Edge-Kernel ( n p , G ) , P ( y | n p , G ) = softmax y ( Linear ( v p )) . The edge kernel of KGAT conducts a hierarchical attention mechanism to propagate information between nodes. It uses token level attentions to produce node representations and sentence level attentions to propagate information along edges. Token Level Attention. The token level attention uses kernels to get the fine-grained representation  z q  p of neighbor node n q , according to node n p . The content propagation and the attention are controlled by kernels. To get the attention weight  q  p i for i -th token in n q , we first conduct a translation matrix M q  p between q -th node and p -th node. Each element of the translation matrix M q  p ij in M q  p is the cosine similarity of their corresponding tokens' BERT representations: M q  p ij = cos( H qi , H pj ) . Then we use K kernels to extract the matching feature (cid:126)K ( M q  p i ) from the translation matrix M q  p (Xiong et al., 2017; Dai et al., 2018; Qiao et al., 2019; MacAvaney et al., 2019): Each kernel K k utilizes a Gaussian kernel to extract features and summarizes the translation score to support multi-level interactions: The attention weights are used to combine the token representations (  z q  p ): Sentence Level Attention. The sentence level attention combines neighbor node information to node representation v p . The aggregation is done by a graph attention mechanism, the same with previous work (Zhou et al., 2019). It first calculate the attention weight  q  p of n q node according to the p -th node n p :  q  p = softmax q ( MLP ( z p   z q  p )) , (11) where  denotes the concatenate operator and z p is the initial representation of n p . It updates the node representation with its neighbors, and the updated information are selected first by the token level attention (Eq. 9) and then the sentence level attention (Eq. 11). Sentence Level Claim Label Prediction. The updated p -th node representation v p is used to calculate the claim label probability P ( y | n p ) : P ( y | n p , G ) = softmax y ( Linear ( v p )) . The prediction of the label probability for each node is also conditioned on the entire graph G , as the node representation is updated by gather information from its graph neighbors. The per-node predictions are combined by the readout function in graph neural networks (Zhou et al., 2019), where KGAT uses node kernels to learn the importance of each evidence. It first uses node kernels to calculate the readout representation  ( n p ) for each node n p :  ( n p ) = Node-Kernel ( n p ) . Similar to the edge kernels, we first conduct a translation matrix M c  e p between the p -th claim and evidence, using their hidden state set H p 1: m and H pm +1: m + n . The kernel match features (cid:126)K ( M c  e p i ) on the translation matrix are combined to produce the node selection representation  ( n p ) :  ( n p ) = 1 m  m (cid:88) i =1 (cid:126)K ( M c  e p i ) . KGAT leverages the kernels multi-level soft matching capability (Xiong et al., 2017) to weight the node-level predictions in the evidence graph based on their relevance with the claim: P ( y | G ) = l (cid:88) p =1 P ( y | n p , G ) P ( n p | G ) . using the ground truth verification label y  . This section describes the dataset, evaluation metrics, baselines, and implementation details in our experiments. Dataset. A large scale public fact verification dataset FEVER (Thorne et al., 2018a) is used in our experiments. The FEVER consists of 185,455 annotated claims with 5,416,537 Wikipedia documents from the June 2017 Wikipedia dump. All claims are classified as SUPPORTS, REFUTES or NOT ENOUGH INFO by annotators. The dataset partition is kept the same with the FEVER Shared Task (Thorne et al., 2018b) as shown in Table 1. Evaluation Metrics. The official evaluation metrics 2 for claim verification include Label Accuracy (LA) and FEVER score. LA is a general evaluation metric, which calculates claim classification accuracy rate without considering retrieved evidence. The FEVER score considers whether one complete set of golden evidence is provided and better reflects the inference ability. We also evaluate Golden FEVER (GFEVER) scores, which is the FEVER score but with golden evidence provided to the system, an easier setting. Precision, Recall and F1 are used to evaluate evidence sentence retrieval accuracy using the provided sentence level labels (whether the sentence is evidence or not to verify the claim). Baselines. The baselines include top models during FEVER 1.0 task and BERT based models. Three top models in FEVER 1.0 shared task are compared. Athene (Hanselowski et al., 2018) and UNC NLP (Nie et al., 2019a) utilize ESIM to encode claim evidence pairs. UCL MRG (Yoneda et al., 2018) leverages Convolutional Neural Network (CNN) to encode claim and evidence. These three models aggregate evidence by attention mechanism or label aggregation component. The BERT based models are our main baselines, they significantly outperform previous methods without pre-training. BERT-pair, BERT-concat and GEAR are three baselines from the previous 2 https://github.com/sheffieldnlp/ fever-scorer Split SUPPORTED REFUTED NOT ENOUGH INFO Train 80,035 29,775 35,639 Dev 6,666 6,666 6,666 Test 6,666 6,666 6,666 Table 1: Statistics of FEVER Dataset. work (Zhou et al., 2019). BERT-pair and BERT-concat regard claim-evidence pair individually or concatenate all evidence together to predict claim label. GEAR utilizes a graph attention network to extract supplement information from other evidence and aggregate all evidence through an attention layer. Soleimani et al. (2019); Nie et al. (2019b) are also compared in our experiments. They implement BERT sentence retrieval for a better performance. In addition, we replace kernel with dot product to implement our GAT version, which is similar to GEAR, to evaluate kernel's effectiveness. Document retrieval. The document retrieval step retrieves related Wikipedia pages and is kept the same with previous work (Hanselowski et al., 2018; Zhou et al., 2019; Soleimani et al., 2019). For a given claim, it first utilizes the constituency parser in AllenNLP (Gardner et al., 2018) to extract all phrases which potentially indicate entities. Then it uses these phrases as queries to find relevant Wikipedia pages through the online Me-diaWiki API 3 . Then the convinced article are reserved (Hanselowski et al., 2018). Sentence retrieval. The sentence retrieval part focuses on selecting related sentences from retrieved pages. There are two sentence retrieval models in our experiments: ESIM based sentence retrieval and BERT based sentence retrieval. The ESIM based sentence retrieval keeps the same as the previous work (Hanselowski et al., 2018; Zhou et al., 2019). The base version of BERT is used to implement our BERT based sentence retrieval model. We use the [CLS] hidden state to represent claim and evidence sentence pair. Then a learning to rank layer is leveraged to project [CLS] hidden state to ranking score. Pairwise loss is used to optimize the ranking model. Some work (Zhao et al., 2020; Ye et al., 2020) also employs our BERT based sentence retrieval in their experiments. batch size to 4 and accumulate step to 8. All models are evaluated with LA on the development set and trained for two epochs. The training and development sets are built with golden evidence and higher ranked evidence with sentence retrieval. All claims are assigned with five pieces of evidence. The BERT (Base), BERT (Large) and RoBERTa (Liu et al., 2019) are evaluated in claim verification. In our experiments, the max length is set to 130. All models are implemented with PyTorch. BERT inherits huggingface's implementation 4 . Adam optimizer is used with learning rate = 5e-5 and warm up proportion = 0.1. The kernel size is set to 21, the same as previous work (Qiao et al., 2019). The experiments are conducted to study the performance of KGAT, its advantages on different reasoning scenarios, and the effectiveness of kernels. The fact verification performances are shown in Table 2. Several testing scenarios are conducted to compare KGAT effectiveness to BERT based baselines: BERT (Base) Encoder with ESIM retrieved sentences, with BERT retrieved sentences, and BERT (Large) Encoder with BERT retrieved sentences. Compared with baseline models, KGAT is the best on all testing scenarios. With ESIM sentence retrieval, same as the previous work (Zhou et al., 2019; Hanselowski et al., 2018), KGAT outperforms the graph attention models GEAR and our GAT on both development and testing sets. It illustrates the effectiveness of KGAT among graph based reasoning models. With BERT based sentence retrieval, our KGAT also outperforms BERT (Base) (Soleimani et al., 2019) by almost 1% FEVER score, showing consistent effectiveness with different sentence retrieval models. When using BERT (Large) as the encoder, KGAT also outperforms the corresponding version of Soleimani et al. (2019). KGAT with RoBERTa performs the best compared with all previously published research on all evaluation metrics. CorefBERT (Ye et al., 2020) extends our KGAT architecture and explicitly models co-referring relationship in context for better performance. The sentence retrieval performances of ESIM and BERT are compared in Table 3. The BERT sentence retrieval outperforms ESIM sentence retrieval significantly, thus also helps improve KGAT's reasoning accuracy. Nevertheless, for more fair comparisons, our following experiments are all based on ESIM sentence retrieval, which is the one used by GEAR, our main baseline (Zhou et al., 2019). The verifiable instances are separated (except instances with NOT ENOUGH INFO label ) into two groups according to the golden evidence labels. If more than one evidence pieces are required, the claim is considered as requiring multi-evidence reasoning. The single evidence reasoning set and the multiple evidence reasoning set contain 11,372 (85.3%) and 1,960 (14.7%) instances, respectively. We also evaluate two additional KGAT variations: KGAT-Node which only uses kernels on the node, with the edge kernels replaced by standard dot-production attention, and KGAT-Edge which only uses kernels on the edge. The results of these systems on the two scenarios are shown in Table 4. KGAT-Node outperforms GAT by more than 0.3% on both single and multiple reasoning sce-Reasoning Model LA GFEVER FEVER Multiple GEAR 66.38 n.a. 37.96 -0.25% GAT 66.12 84.39 38.21 KGAT-Node 65.51 83.88 38.52 0.31% KGAT-Edge 65.87 84.90 39.08 0.87% KGAT-Full 65.92 85.15 39.23 1.02% Single GEAR 78.14 n.a. 75.73 -1.69% GAT 79.79 81.96 77.42 KGAT-Node 79.92 82.29 77.73 0.31% KGAT-Edge 79.90 82.41 77.58 0.16% KGAT-Full 80.33 82.62 78.07 0.65% Table 4: Claim Verification Accuracy on Claims that requires Multiple and Single evidence Pieces. narios. As expected, it does not help much on GFEVER, because the golden evidence is given and node selection is not required. It illustrates KGAT-Node mainly focuses on choosing appropriate evidence and assigning accurate combining weights in the readout. KGAT-Edge outperforms GAT by more than 0.8% and 0.1% on multiple and single evidence reasoning scenarios, respectively. Its effectiveness is mostly on combining the information from multiple evidence pieces. The multiple and single evidence reasoning scenarios evaluate the reasoning ability from different aspects. The single evidence reasoning mainly focuses on selecting the most relevant evidence and inference with single evidence. It mainly evaluates model de-noising ability with the retrieved evidence. The multiple evidence reasoning is a harder and more complex scenario, requiring models to summarize necessary clues and reason over multiple evidence. It emphasizes to evaluate the evidence interactions for the joint reasoning. KGAT-Node shows consistent improvement on both two reasoning scenarios, which demonstrates the important role of evidence selection. KGAT-Edge, on the other hand, is more effective on multiple reasoning scenarios as the Edge Kernels help better propagate information along the edges. More Concentrated Attention. This experiment studies kernel attentions by their entropy, which reflects whether the learned attention weights are focused or scattered. The entropy of the kernel attentions in KGAT, the dot-product at-(a) Edge Attention. tentions in GAT, and the uniform attentions are shown in Figure 3. The entropy of Edge attention is shown in Figure 3(a). Both GAT and KGAT show a smaller entropy of the token attention than the uniform distribution. It illustrates that GAT and KGAT have the ability to assign more weight to some important tokens with both dot product based and kernel based attentions. Compared to the dot-product attentions in GAT, KGAT's Edge attention focuses on fewer tokens and has a smaller entropy. The entropy of Node attentions are plotted in Figure 3(b). GAT's attentions distribute almost the same with the uniform distribution, while KGAT has concentrated Node attentions on a few evidence sentences. As shown in the next experiment, the kernel based node attentions focus on the correct evidence pieces and de-noises the retrieved sentences, which are useful for claim verification. More Accurate Evidence Selection. This experiment evaluates the effectiveness of KGAT-Node through attention distribution and evidence recall. The results are shown in Figure 4. We first obtain the node attention score in the evidence graph from KGAT or GAT, and calcu-(a) GAT. late the statistics of the maximum one for each claim, as most of which only require single evidence to verify. The attention score of the highest attended evidence node for each claim is plotted in Figure 4(a). As expected, KGAT concentrates its weight to select evidence nodes and provides a focused attention. Then the evidence selection accuracy is evaluated by their evidence recall. We first rank all evidence pieces for each claim. Then the evidence recall with different ranking depths is plotted in Figure 4(b). KGAT achieves a much higher recall on top ranking positionsonly the first ranked sentence covers nearly 80% of ground truth evidence, showing the node kernels' ability to select correct evidence. This also indicates the potential of the node kernels in the sentence retrieval stage, which we reserve for future work as this paper focuses on the reasoning stage. Fine-Grained Evidence Propagation. The third analysis studies the distribution of KGAT-Edge's attention which is used to propagate the evidence clues in the evidence graph. Figure 5 plots the attention weight distribution of the edge attention scores in KGAT and GAT, one from kernels and one from dot-products. The kernel attentions again are more concentrated: KGAT focuses fewer words while GAT's dot-product attentions are almost equally distributed among all words. This observation of the scattered dot-product attention is consistent with previous research (Clark et al., 2019). As shown in the next case study, the edge kernels provide a fine-grained and intuitive attention pattern when combining evidence clues from multiple pieces. Table 5 shows the example claim used in GEAR (Zhou et al., 2019) and the evidence sentences retrieved by ESIM, among which the first two are required evidence pieces. Figure 6 presents the distribution of attentions from the first evidence to the tokens in the second evidence (  2  1 i ) in KGAT (Edge Kernel) and GAT (dot-product). The first evidence verifies that Al Jardine is an American musician but does not enough information about whether Al Jardine is a rhythm guitarist. The edge kernels from KGAT accurately pick up the additional information evidence (1) required from evidence (2): rhythm guitarist. It effectively fills the missing information and completes the reasoning chain. Interesting, Al Jardine also receives more attention, which helps to verify if the information in the second evidence is about the correct person. This kernel attention pattern is more intuitive and effective than the dot-product attention in GAT. The later one scatters almost uniformly across all tokens and hard to explain how the joint reasoning is conducted. This seems to be a common challenge of the dot-product attention in Transformers (Clark et al., 2019). This paper presents KGAT, which uses kernels in Graph Neural Networks to conduct more accurate evidence selection and fine-grained joint reasoning. Our experiments show that kernels lead to the more accurate fact verification. Our studies illustrate the two kernels play different roles and contribute to different aspects crucial for fact verification. While the dot-product attentions are rather scattered and hard to explain, the kernel-based attentions show intuitive and effective attention patterns: the node kernels focus more on the correct evidence pieces; the edge kernels accurately gather the necessary information from one node to the other to complete the reasoning chain. In the future, we will further study this properties of kernel-based attentions in neural networks, both in the effectiveness front and also the explainability front. This research is jointly supported by the NSFC project under the grant no. 61661146007, the funds of Beijing Advanced Innovation Center for Language Resources (No. TYZ19005), and the NExT++ project, the National Research Foundation, Prime Ministers Office, Singapore under its IRC@Singapore Funding Initiative.
abstain method abstain abstain method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result objective other other other other other other other other other other other other other other other other other other other abstain abstain abstain abstain abstain method abstain abstain abstain abstain other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method result objective abstain method other