State-of-the-art abstractive summarization systems often generate hallucinations ; i.e., content that is not directly inferable from the source text. Despite being assumed incorrect, we find that much hallucinated content is factual, namely consistent with world knowledge. These factual hallucinations can be beneficial in a summary by providing useful background information. In this work, we propose a novel detection approach that separates factual from non-factual hallucinations of entities. Our method utilizes an entity's prior and posterior probabilities according to pre-trained and finetuned masked language models, respectively. Empirical results suggest that our approach outperforms five baselines and strongly correlates with human judgments. Furthermore, we show that our detector, when used as a reward signal in an off-line reinforcement learning (RL) algorithm, significantly improves the factuality of summaries while maintaining the level of abstractiveness. 1 1 Introduction State-of-the-art abstractive summarization systems can generate fluent summaries with high automatic evaluation scores in terms of ROUGE (Lin, 2004). However, recent studies have shown that these systems are prone to hallucinate content that is not supported by the source document (Maynez et al., 2020; Kang and Hashimoto, 2020; Durmus et al., 2020; Zhao et al., 2020; Filippova, 2020; Kryscinski et al., 2020). For instance, Maynez et al. (2020) discovered that 64.1% of the summaries generated by a BERT-based abstractive summarization model on XSUM (Narayan et al., 2018a) contain hallucinations. Previous studies commonly assume that hallucination is an undesirable behavior in abstractive summarization systems. They investigate the 1 https://github.com/mcao516/EntFA Source : Under the proposals, 120,000 additional asylum seekers will be distributed among EU nations, with binding quotas. (...) Mr Juncker told the European Parliament it was not a time to take fright. (...) He said tackling the crisis was a matter of humanity and human dignity. It is true that Europe cannot house all the misery in the world. But we have to put it into perspective. (...) Generation : European Commission President Jean-Claude Juncker has set out his proposals for dealing with the migrant crisis in a speech to MEPs, saying Europe cannot house all the misery in the world. cause of model hallucination (Kang and Hashimoto, 2020; Wang and Sennrich, 2020) and propose methods that reduce the frequency of all hallucinations (Filippova, 2020; Zhao et al., 2020; Nan et al., 2021; Narayan et al., 2021). Our stance in this paper is that hallucinations are not always undesirable : many factual hallucinations provide additional world knowledge that is important for summary comprehension. Table 1 presents one such example from XSUM : the hallucinated content European Commission President provides additional background information on the role of Mr. Juncker . Factual hallucinations refer to content that is verifiable by world knowledge but not inferable from source text. We thus argue that not all hallucinations should be treated equally; in particular, factual hallucinations may be less deleterious or even potentially beneficial to to be included in a summary, as opposed to non-factual ones. We propose a method to classify entities according to whether they are hallucinations and whether they are factual (if hal-lucinated). We focus on entities (e.g., persons, locations, dates, cardinal numbers) because they are necessary to express the most salient pieces of in-3340 formation in a summary. Moreover, entity hallucinations are common in generated summaries. As we will show later in our work, about 30% of entities generated by BART (Lewis et al., 2020) on XSUM test set are hallucinated. Our approach is inspired by the observation that many hallucinated entities are generated with low probabilities. This observation suggests that the summarization model's confidence correlates with the factuality statuses of generated entities. In other words, the uncertainty is indicative of the likelihood of whether generated entities are hallucinated and non-factual. We refer to the probability of an entity being in a summary without considering the source document as its prior probability, and its probability given the document as its posterior probability. Our as-sumption is that if an entity in a generated summary results in a factual error, giving the source should not provide more evidence for it, resulting in a small change in probability between the prior and the posterior. Based on this assumption, we propose to use the prior and posterior probabilities as the key features in a simple classifier that predicts an entity's hallucination status and factuality. Due to the lack of fine-grained hallucination annotation, we create an entity-level hallucination and factuality annotation on the XSUM dataset. We evaluate our detection method on this annotated dataset as well as annotations from Maynez et al. (2020). On both datasets, our approach outperforms five baseline models at identifying nonfactual hallucinations. In addition, our approach has a strong correlation with the factuality scores given by human judges. Besides, we show that our detector, when used as a reward signal in training neural-based summarizers with the off-line RL algorithm, significantly improves the factuality of generated summaries even when the underlying dataset is noisy. Our contributions are the following: ( i ) We demonstrate that an entity's prior and posterior probabilities can be used to infer whether it is hallucinated and factual. Based on this hypothesis, we propose a novel approach for entity-level hallucination detection and factuality checking. Our approach outperforms five baselines from previous work on two human-annotated datasets, in addition to having a strong correlation with summary-level factuality scores given by human judges. ( ii ) We empirically demonstrate that our classifier can provide reliable reward signals for RL algorithms, leading to improved factuality while maintaining the level of abstractiveness in generated summaries. ( iii ) We create a set of entity-level hallucination annotations. The correctness of summarization systems' outputs has been evaluated as one aspect of content selection in the past, for example using the Pyramid method (Nenkova and Passonneau, 2004). As neural abstractive summarizers have become popular, their issues with correctness have sparked much recent work that focus specifically on model hallucinations and summary factuality (Kryscinski et al., 2020). Maynez et al. (2020) conducted a large-scale human evaluation of several neural abstractive summarization systems, and found that hallucinations are common among the outputs of different summarization models. Recently, many methods have been proposed to reduce model hallucination. Kang and Hashimoto (2020) propose a loss truncation training algorithm that filters out noisy training samples which may lead a model to hallucinate. Zhao et al. (2020) use a verification system to recognize non-factual quantities in summaries and adopt a re-ranking system to reduce the number of hallucinated quantities in the final output summary. Narayan et al. (2021) use entity chains to mitigate the hallucination problem in the generation of abstractive summaries. Nan et al. (2021) show that data filtering and use a summary-worthy entity classification task as an auxiliary training objective can help improve model's entity-level factuality. Filippova (2020) proposed a method for controlling hallucination in data-to-text generation task. They suggest that a conditional language model (CLM) will put more probability mass on a nonhallucinated entity than an unconditional language model (LM). Our work differs in that we focus on both hallucination and factuality. Also, our method works at the entity-level rather than the sentence-level, and is geared towards text summarization. Another line of work focuses on evaluating the factual consistency of abstractive summarization systems. Kryscinski et al. (2020) train models on an artificially corrupted dataset for factual errors detection. Cao et al. (2020) induce artificial perturbations in text to train a summary error correction system, but find that there is a large gap between such artificial perturbations and the type of hallucinations that are generated by abstractive summarizers. (Goyal and Durrett, 2020) measure factual consistency by checking whether the semantic relationship manifested by individual dependency arcs in the generated summary is supported by the source document. Wang et al. (2020); Dong et al. (2020); Durmus et al. (2020) measure and improve the factual consistency of summaries by asking and answering questions based on generated summaries and input documents. In this section, we propose a novel detection approach that separates factual from non-factual hallucinations of entities (Section 3.2), and present a factuality-aware training framework for summarization models trained on noisy dataset (Sec-tion 3.3). Let ( S, R ) be a pair of a source document and a reference summary, where S = ( s 1 , ..., s M ) is the source document with M tokens, and R = ( r 1 , ..., r L ) is the reference summary with L tokens. Let G = ( g 1 , ..., g N ) be the model-generated summary with N tokens. For each named entity e k , which we assume to be a span of tokens g i k , ..., g i k + | e k | 1 ( | e k |  1) starting at position i k in G , the task is to determine whether e k is hallucinated, and whether it is factual. We define an entity as hallucinated if it is not directly inferable in its generated context given the input document S . If an entity is hallucinated, we further classify it into two subtypes: factual hallucinations and non-factual hallucinations . Factual hallucinations cannot be directly entailed in their generated context from the source document but can be based on world knowledge (see Table 1). Non-factual hallucinations are entities that are neither inferable from the source nor based on world knowledge. We now define the prior and posterior probabilities of an entity, which we will use to predict its hallucination and factuality statuses. For entity e k , we define its prior probability p prior ( e k ) as the probability of its generation by a language model that does not have access to the source text. If e k spans multiple tokens, we compute its probability auto-regressively. Let c k be the context of entity e k in G , excluding the tokens in e k . Then: p prior ( e k ) = fP MLM ( e k | c k ) (1) = | e k | (cid:89) t =1 PMLM ( e tk | e 1 ...t  1 k , c k ) (2) which we compute using a masked language model PMLM . The posterior probability p pos ( e k ) of entity e k is the conditional probability of the entity given the context and the source text: p pos ( e k ) = PCMLM ( e k | c k , S ) (3) = | e k | (cid:89) t =1 PCMLM ( e tk | e 1 ...t  1 k , c k , S ) , (4) where CMLM is a conditional masked language model. CMLM is an encoder-decoder model that is trained with a masked language model objective on a parallel dataset. Specifically, a CMLM predicts a target sequence T given a source text S and part of the target T masked , where T masked is the target sequence with a random entity being masked. In order to correctly generate the missing part of the sentence, the model needs to condition on both T masked and S . Alternatively, we can calculate the entity's posterior probability using a conditional language model (CLM) instead of a CMLM. In this case, the entity's posterior probability is defined as PCLM ( e k | c e k , S ) where c e k = g 1 , ..., g i  1 . Note that CLM is only conditioned on the left context. Training a Discriminator To classify the hallucination and factuality statuses of a given entity, we need to train a discriminator model. We use the K-Nearest Neighbors (KNN) algorithm since it requires no training and makes minimal assumptions about the form of the decision boundary, as a non-parametric method. It also offers adequate interpretability. The KNN classifier is trained using the prior and posterior probabilities as features on our labeled dataset. Since the classifier is used for entity hallucination and factuality assessment, we refer to it as ENTFA . Besides using 3342 the prior/posterior probability as features, we also add a binary overlap feature that indicates whether the entity appears in the document. We train two classifiers for hallucination detection and factuality checking tasks respectively. We now propose a factuality-aware training approach for summarization systems that combines our factuality assessment model with the latest offline RL technique. RL for Text Generation Sequence generation of the tokens in the summary text can be viewed as a finite Markov Decision Process (MDP). At each time-step t , the state s t consists of the source text x and the previously generated tokens y <t , s t = ( y <t , x ) . The agent, which is the summarization model, takes an action by generating a new token a t . Depending on the action taken, the agent gets a reward r t = R ( s t , a t ) and deterministically transitions to the next state s t +1 = ( y <t +1 , x ) . The probability of each action (i.e., token) is specified by the policy   ( a t | s t ) . The goal of the agent is to maximize the discounted cumulative reward throughout the trajectory: J (  ) = E    (cid:104) (cid:80) Tt =0  t r t (cid:105) . When training the summarization model with human-written reference summaries, we can frame the training process as an off-line RL problem with expert demonstrations (i.e., the reference sum-maries). In this setting, since we are sampling trajectories from a behavior policy, we need an importance sampling term w t to correct the gradient estimation. Following Pang and He (2021)'s work, we approximate w t with   ( a t | s t ) and this gives us the following gradient expression of the objective function:   J (  ) = E    b (cid:104) (cid:88) t =0   ( a t | s t )   log   ( a t | s t )  Q ( a t , s t ) (cid:105) (5) where  Q ( a t , s t ) = (cid:80) Tt  = t  t   t r t  is the estimated return from state s t and  b is the behavior policy from which we draw trajectories  . In our case,  b is the (noisy) summarization dataset. Training with a Factuality-based Reward One problem in the off-line RL setting is that expert demonstrations, which in our case are the reference summaries, are often noisy and contain content that cannot be inferred from the source document. The commonly used teacher forcing training encourages the model to blindly imitate the training data, which leads to model hallucination at inference time (Kang and Hashimoto, 2020). To discourage the model from overfitting to the noise in the training set, we use the predictions from our classifier as factuality reward signals to guide the training of the summarization model. In the off-policy learning stage, we use our factuality classifier to label all the entities in the training set. If an entity is classified by our classifier as non-factual, we consider it noise and give it a negative reward  r nfe . For factual entities and other tokens, we use the posterior probability from a MLE-trained model as token-level rewards, as in (Pang and He, 2021). Formally, we have: R ( s t , a t ) = (cid:40)  r nfe , if a t is non-factual p MLE ( a t | s t ) , otherwise 4 Dataset 4.1 XENT dataset To study entity hallucination and factuality in abstractive summarization, we need annotations of entityor token-level hallucination. To the best of our knowledge, there is no such dataset available. Therefore, we create a dataset ourselves, which we call the XENT dataset. We 2 annotate 800 summaries generated by BART, which is one of the state-of-the-art abstractive summarization models. The input documents are randomly selected from XSUM test set. We choose XSUM because it is more abstractive than other summarization datasets. We extract 2,838 entities from the 800 generated summaries. We randomly select 30% of the samples as our test set. We manually labeled each entity with one of the following three tags: non-hallucinated, factual hallucination, and non-factual hallucination. First, we extract entities from the given summary using automatic NER tools (Honnibal and Montani, 2017). Then, we check whether each property associated with the identified entity can be directly entailed using the information from the source document. If so, then the property is non-hallucinated. For instance, consider the entity European Commission President Jean-Claude Juncker in Table 1. The last name Juncker can be directly entailed from 2 Two coauthors and three graduate students. the source document. Therefore, it is not a hallucination. However, the first name Jean-Claude and the position information European Commission President are not mentioned in the source. In the next step, we need to decide whether these information is factual or not using world knowledge. This often requires external resources such as Wikipedia or Google Search. In this case, European Commission President and Jean-Claude are both factual. If there is no information found online to prove or disprove the hallucinated entity, it is labeled as non-factual. There is a special case where the entity misrepresents information from the document. For instance, the summary might include a number from the document but that number is actually related to a different event. In this case, the entity is considered as an intrinsic hallucination (Maynez et al., 2020). In this work, we will focus on extrinsic hallucinations, so we discarded all intrinsic hallucinations in our experiments. Table 3 shows the distribution of entities by hallucination and factuality status in our labeled dataset. We show an example for each hallucination type in Table 2. Inter-Annotator Agreement We report Fleiss's Kappa (  ) to access the reliability of agreement between annotators. We compute agreement on a subset of 800 entities and obtain almost perfect agreement ( 0 . 80    1 . 00 ) with  = 0 . 809 . Following Pagnoni et al. (2021), we also report the percentage  of annotators that agree with the majority class. We obtain  = 0 . 931 of annotators agreeing with the majority class on the four-category annotation which shows substantial agreement. Recently, Maynez et al. (2020) released a set of factuality and hallucination annotations for XSUM . For each generated summary, they labeled the hallucinated spans as well as the overall factuality of the summary. Compared with our labeling approach, their annotation has a lower granularity and does not distinguish between factual and non-factual hallucination. Therefore, we have to convert their dataset first before using it for evaluation. To perform entity-level factuality checking on their dataset, we do the following: First, we extract entities from the annotated summaries. For entities that are extracted from factual summaries, we label them as factual entities. For each entity from non-factual summary, if it is inside an extrinsic hallucinated span, then we assume the entity is non-factual. Otherwise the entity is labeled as a factual. This process gives us a new dataset that has the same format as ours for entity-level factuality evaluation. We refer to this new dataset as the MENT dataset. However, it is worth pointing out that the converted dataset is noisy. For instance, in Maynez et al. (2020)'s annotation, the entire generated summary is often labeled as a hallucinated span if it 3344 does not capture the meaning of the document well. In this case, the hallucinated span could still contain faithful entities with respect to the source document. This could result in false-positive non-factual entities after the conversion. Therefore, we filter out entities in the extrinsic hallucination span that also appear in the source document. We evaluate our method on entity-level hallucination and factuality classification tasks on XENT and MENT . For each entity in the summary, the model predicts a hallucination label and a factuality label. We will conduct factual and hallucination assessments separately for comparison with the baselines. We compare our method with five baselines models, which are discussed in detail in Section 6.1. In addition to entity-level classification performance, we also evaluate our methods by correlating them against human judgments of factuality. Previous work has collected summary-level judgments of factuality from human annotators, which are then correlated with automatic evaluation measures applied to those summaries. To apply our entity-level method, we use the lowest classifier confidence for the factual class among its entities as the factuality score for the entire summary. We evaluate correlation on two datasets by Pagnoni et al. (2021) and Wang et al. (2020). To evaluate our factuality-aware training approach proposed in Section 3.3, we train a summarization model with factuality rewards and evaluate model's predictions on XSUM test set. To evaluate the faithfulness of generated summaries, we use automatic faithfulness evaluation tools FEQA (Durmus et al., 2020) and DAE (Goyal and Durrett, 2020) 3 . We also calculate ROUGE scores, and the percentage of n -grams and percentage of entities in the generated summaries that are not found in the source document (ENFS). The percentage of novel n -grams 3 In this work, we define the faithfulness of the summary as whether it is faithful with respect to the source. reflects the extractiveness of summarization model. Training CMLM & MLM For training the CMLM, we use both XSUM , Narayan et al. (2018b)) and the CNN/Dailymail dataset (Hermann et al., 2015) dataset. To build a training corpus for CMLM, we randomly select one entity in each reference summary and mask it with a special [ MASK ] token. We append a [ S ] token at the beginning of each summary. The document and summary are concatenated together (separated by [ \S ] token) as CMLM's input. The training target is the reference summary without any masking. If there is no specification, we use the CMLM trained on XSUM . For the MLM, we use the large BART model. BART is pre-trained on five different reconstruction tasks including token masking and text infilling. For more experimental setup and hyper-parameter setting details, see Appendix A.3. Baselines We compare with five baseline methods: (1) The overlap-based method checks the word overlap between the summary and the source document. In our case, we check whether a given entity in the generated summary also exist in the source document. If it does not, the entity is clas-3345 sified as both hallucinated and non-factual. (2) The synonym-based baseline extends the overlap-based baseline by checking the overlap of summary synonyms and source synonyms. See Zhou et al. (2020) for more details. (3) The alignment-based baseline is based on the unsupervised word alignment method SimAlign by Jalili Sabet et al. (2020). SimAlign extracts word alignments from similarity matrices induced from pretrained embed-dings. In our task, we treat all unaligned entities in summaries as hallucinated and non-factual. (4) The LM-based method is proposed by Filippova (2020). The LM-based method uses LM and CLM to compute the token's prior and posterior probability. In Filippova (2020)'s work, they compare the value of p prior and p pos . If the generated token does not match the reference and p prior is greater than p pos , the token is classified as hallucinated. Since we are evaluating the generated summary but not the reference, we modify their method to the following: if the entity is not found in the source and p prior > p pos , then the entity is classified as non-factual and hallucinated. (5) Zhou et al. (2020) frame the hallucination detection task as a sequence labeling task. They train a hallucination labeling model on synthetic data. We adapt their model to our task by finetuning their model on XENT . Evaluation Results on XENT Table 4 shows the evaluation results of our classifiers and baselines in terms of both entity factuality and hallucination status classification. The results show that our approach outperforms five baselines on the factuality classification task. To show that our model is statistically better than the baselines, we run a 10 -fold cross-validated paired t-test comparing our model with five baselines. The results show that our model is better than the baseline models with p -value less than 3 . 27 e  5 . On the hallucination detection task, the overlap-based and synonym-based baselines achieve relatively high accuracy. However, these methods cannot distinguish between factual and non-factual hallucinations. This is the reason for their performance degradation on factuality classification task. For hallucination classification, the reason computing word overlap with the source does not completely solve the hallucination detection problem is that hallucination is defined based on the semantic relationship between the source and the summary. There can exist words that are not in the source document but which can nevertheless be inferred from it. shows the evaluation results on MENT . ENTFA are learned on our annotated training set with k set to 20. The performance of all models is lower on this dataset. This may be due to fact that the converted dataset is noisier than the XENT dataset (see Section 4.2). For the factuality classification task, our model outperforms five baseline models. This demonstrates the generalizability of our approach. Table 6 presents the correlation evaluation results. On Pagnoni et al. (2021)'s benchmark dataset, our approach has the highest partial Pearson correlation coefficient  = 0.183 ( p < 1 e  8 ). On Wang et al. (2020)'s dataset (right column), our approach outperforms all other automatic metrics significantly. These results indicate that our model can be used for automatic factuality evaluation of summaries at both the entity and sentence levels. Baselines We compare our approach with four baselines: a teacher forcing trained summarizer (MLE), a RL-based summarizer (RL) (Pang and He, 2021) and a summarizer trained with the loss truncation technique from Kang and Hashimoto (2020). We also replace our factuality assessment model ENTFA with Filippova (2020)'s approach (LM-based) for entity factuality labeling as another baseline model (see Section 3.3). Table 7 shows the evaluation results on XSUM . The results show that our approach outperforms all baselines with fewer non-factual entities and higher faithfulness scores. Note that our approach has the lowest ENFS rate while having the highest percentage of factual hallucinations. Compared with the loss truncation baseline, our method also produces more novel n -grams. These show that our method does not improve the factuality of the model by simply making the model more extractive. Figure 1 shows the factuality and abstractiveness trade-off curves of our model compared to the loss truncation baseline. At the same level of ROUGE performance, our method can obtain a higher factuality score. This further proves that our model can generate both factual and high-quality summaries compared with the loss truncation baseline. To explore the effect of each feature, we conduct an ablation study by training the KNN classifier with fewer features. The results are illustrated in Table 8 and show that all the proposed features are useful. For factuality classification, The performance w/o posterior drops significantly from 81.82 to 70.30. This result suggests that the posterior probability is crucial for factuality classification. For hallucination classification, the overlap-based feature has the most significant impact on model performance. Figure 2 plots entities in the XENT dataset according to their prior and posterior probabilities and shows the KNN classification boundaries of ENTFA w/o overlap. In Figure 2a, we find that the non-factual hallucinated entities are clustered around the origin. This is in line with our expectations since non-factual hallucinations have lower prior and posterior probabilities. Both factual hallucinated and non-hallucinated entities are gathered in the top area with high posterior probabilities. In Figure 2b, the KNN classifier separates the factual and non-factual entities with clear boundaries. A large part of the factual hallucinated entities are correctly identified by CMLMXSUM with relatively high posterior probabilities. This explains our model's superior performance on factuality checking. The top and right histograms in Figure 2b show the entity distribution over prior and posterior probability value respectively. As shown in 2b's histogram, factual entities have significantly higher posterior probability than that of non-factual entities on average. Figure 3 shows histograms of the prior and posterior probabilities of entities from MLM and CMLMXSUM , separated by their class (i.e., whether they are hallucinated and/or factual). Nonhallucinated entities have higher posterior probability than factual and non-factual hallucinations on average. In this paper, we investigate the hallucination and factuality problems in abstractive summarization. We show that about 30% of entities generated by state-of-the-art summarization model are hallucinated. More interestingly, more than half of the hallucinated entities are factual with respect to the source document and world knowledge. We propose a novel method based on the entity's prior and posterior probabilities according to masked language models. Our approach outperforms five base-3347 ROUGE % of novel n-gram Faithfulness ENTFA System R1  RL  unigrams  bigrams  % ENFS  FEQA  DAE  % Factual Ent  % Factual Hal  MLE 45.1 37.3 27.86 74.47 42.0 25.9 34.6 82.8 21.4 RL 45.8 37.6 28.14 74.73 43.2 25.6 33.3 82.8 21.6 LM-based 43.2 34.6 29.75 75.86 38.2 24.2 31.3 87.4 21.7 Loss trunc (c=0.3) 44.1 36.0 26.82 73.39 41.3 26.3 36.4 83.9 21.3 Loss trunc (c=0.7) 42.7 34.8 26.61 73.19 40.6 26.7 38.8 84.1 20.7 Ours ( r nfe = 2 . 0 ) 44.6 36.2 27.71 74.90 37.2 26.5 37.3 90.1 24.0 Ours ( r nfe = 4 . 0 ) 43.0 34.9 26.87 74.11 32.8 27.3 40.8 92.5 22.4 Table 7: Comparison of different summarization models. line models on both factuality classification and hallucination detection tasks on human-annotated datasets. In addition, using our classifier as a reward signal vastly improves the factuality of summarization systems. Our approach is limited to entity-level hallucination and factuality classification. In the future, we are interested in extending our work to arbitrary text spans. This research was supported by the Canada CIFAR AI Chair program and Samsung Electronics. We would also like to thank Compute Canada for providing us computing resources.
abstain result abstain objective method result result abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain method abstain abstain abstain objective method abstain result method abstain abstain method result objective method method result abstain result objective objective result abstain objective abstain method other other other other other other other other other other objective abstain other other other other other other method abstain abstain method method method abstain abstain method abstain method method abstain method other abstain abstain abstain method abstain abstain method method abstain method method method method method abstain abstain abstain abstain abstain abstain method method method method method abstain method method method method method method method method abstain method method method method method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method method method method method abstain method method abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method method abstain method method method abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result result result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result result abstain result abstain result abstain abstain result result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective result abstain objective other abstain result method objective other other