Modern summarization models generate highly fluent but often factually unreliable outputs. This motivated a surge of metrics attempting to measure the factuality of automatically generated summaries. Due to the lack of common benchmarks, these metrics cannot be compared. Moreover, all these methods treat factuality as a binary concept and fail to provide deeper insights on the kinds of inconsistencies made by different systems. To address these limitations, we devise a typology of factual errors and use it to collect human annotations of generated summaries from state-of-the-art summarization systems for the CNN/DM and XSum datasets. Through these annotations we identify the proportion of different categories of factual errors in various summarization models and benchmark factuality metrics, showing their correlation with human judgement as well as their specific strengths and weaknesses. 1 1 Introduction Factuality is defined as a measure of whether eventualities are characterized as corresponding to facts, possibilities, or situations that do not hold in the world (Sauri, 2008; Saur and Pustejovsky, 2012). In summarization, this world is the article, which is taken as ground-truth, and the output summary must be faithful to the article's facts. Despite advancements in neural abstractive summarization (Narayan et al., 2018; Liu and Lapata, 2019; Lewis et al., 2020),  30% of summaries have factual inconsistencies (Cao et al., 2018). With summarization being an integral component of information consumption, this highlights a need for ensuring summarization systems are factually consistent and developing methods for evaluating them. METEOR (Papineni et al., 2002; Lin, 2004; Lavie and Agarwal, 2007)  are insufficient to measure the factual correctness of summaries and fail to correlate with the human judgements of factuality (Falke et al., 2019; Kryscinski et al., 2019). More recent metrics proposed to improve the evaluation of summarization factuality (Kryscinski et al., 2020; Durmus et al., 2020; Wang et al., 2020; Maynez et al., 2020) cannot be compared due to the lack of common benchmarks. More critically, while these approaches differ in the way they model factuality, they all consider factuality as a binary concept, labeling summaries of any length as factual or non-factual. They do not provide any fine-grained understanding of the factual errors made by different systems that could serve as an actionable feedback on a system's limitations. The binary factuality of a text can be difficult to determine. Falke et al. (2019) show relatively low crowdexpert agreement, indicating the presence of subjectivity in the annotation process. Moreover, not all factual errors are equally important and the number of errors can have a significant impact on the perceived factuality of a text. This suggests that non-factuality should be modeled as a multidimensional construct and not a label. In this work, we propose a linguistically motivated typology of factual errors for fine-grained analysis of factuality in summarization systems (2). Our typology is theoretically grounded in frame semantics (Fillmore et al., 1976; Palmer et al., 2005) and linguistic discourse theory (Brown and Yule, 1983). It provides several benefits. First, we find that decomposing the concept of factuality in (relatively) well-defined and grounded categories makes the final binary decision more objective leading to near perfect agreement between crowd and expert annotators (  = 0 . 86 ). Second, this approach provides some measure of the degree of non-factuality both in terms of the quantity and the category of factual violations that appear FRANK Benchmark Data Annotation Per sentence error category annotation Typology of Factual Errors Frame semantics, discourse analysis Summarization System Evaluation Which mistakes? in the text. This typology also provides us with the means to categorize the types of errors made by summarization systems, helping us gain deeper insights than simply categorizing content as factual or hallucinated. We define an annotation protocol of factuality based on our typology and collect a dataset of human judgements over a diverse set of model generated summaries on the CNN/DM (Hermann et al., 2015) and XSum (Narayan et al., 2018) datasets (3). Through this dataset, we aim to both assess the factuality of summarization systems and benchmark recently proposed factuality metrics. In 4 we discuss various state-of-art models and show a detailed analysis of the factual errors they make. Finally, in 5 we evaluate multiple summarization metrics against our benchmark and show their strengths and weaknesses in detecting specific types of factual errors. Figure 1 shows an overview of this work. Previous studies of factuality in summarization only distinguish factual and hallucinated content (Kryscinski et al., 2019; Maynez et al., 2020) and provide limited insights on the fine-grained types of factual errors. In the simplest case, factual errors appear within a single proposition. However, as summaries include several sentences, discourse markers describe relations across propositions. These cross-sentence links, such as causality or temporal ordering, can introduce inconsistencies with the article. Furthermore, information in the summary should be verifiable given the article. This understanding outlines different levels of linguistic structure where factual mistakes can arise in summaries: at the semantic frame level, at the discourse level, or because the content cannot be verified. Below we define a typology of factual errors further detailing these three levels. This typology is theoretically grounded in frame semantics (Fillmore et al., 1976; Baker et al., 1998; Palmer et al., 2005) and linguistic discourse analysis (Brown and Yule, 1983). Examples for each category are shown in Table 1. 2.1 Semantic Frame Errors A semantic frame is a schematic representation of an event, relation, or state, which consists of a predicate and a list of participants, called frame elements (Baker et al., 1998). A semantic frame has both core and non-core frame elements (FE). Core frame elements are essential to the meaning of the frame, while non-core (e.g. location, time) provide additional descriptive information. Our first three categories capture factual errors in each of these components (frame, core and non-core FE) respectively. Predicate Error (PredE): Category PredE encompasses errors where the predicate in a summary statement is inconsistent with the source text. More generally, this represents cases where the frame from a summary statement does not align with what is expressed in the source text. Entity Error (EntE): Category EntE captures errors where the primary arguments (like entities) of the predicate are wrong or have the wrong attributes, although the relation was expressed in the original text. More generally, these account for cases where the core frame elements in a frame are wrong. This also captures directionality errors where the elements are interchanged (similar to agent-patient swap). Circumstance Error (CircE): In additional to the core arguments, predicates can be further speci-fied using additional information or attributes that describe the circumstance in which the arguments Category Description Example PredE Relation Error The predicate in the summary statement is inconsistent with the source article. and predicates interact (e.g. location, time, manner, direction, modality). Category CircE captures errors where one or more such attributes (non-core frame elements within a frame) are wrong. The communicative intent of an author is also expressed through relations that hold between parts of the text. Factual errors in summarized text can often extend beyond a single semantic frame introducing erroneous links between discourse segments. Below we outline such categories of errors which are grounded in discourse analysis and rhetorical structure theory (RST) (Brown and Yule, 1983; Mann and Thompson, 1988). RST is an elaborate system for annotating coherence relations in discourse. Some examples of such relations include: Elaboration, Background, Motivation, and Volitional Cause. Here we depart from semantic frame terminology as its rooting in a single frame does not allow us to represent such errors. Coreference Error (CorefE): Category CorefE accounts for errors where pronouns and other types of references to previously mentioned entities either are incorrect or have no clear antecedents, making them ambiguous. Discourse Link Error (LinkE): Category LinkE encompasses errors involving a discourse link between different statements. These include errors of incorrect temporal ordering or incorrect discourse links (e.g. RST relations, discourse connectors) between statements. Often statements in a summary cannot be verified against the source text due to difficulty in aligning them to the source. Below we outline two categories of errors for such cases. Out of Article Error (OutE): Since summaries of a document should only contain information that can be deduced from the original text, we include a category for such errors OutE (prior work refers to this as extrinsic hallucinations (Maynez et al., 2020)). Grammatical Error (GramE): We use GramE to categorize statements that are not well formed. When grammatical mistakes make the meaning of a statement incomprehensible or ambiguous, it cannot be verified against the source and is thus considered trivially wrong. Minor grammatical errors are acceptable. Finally, for completeness in our annotation exercise, we add two additional categories Others (OthE) for factually errors that do not correspond to any of the above categories and Not an Error (NE) for statements that do not contain any errors. Beyond theoretical grounding, we empirically verify our typology through large scale human annotations annotations of five abstractive summarization models on the CNN/DM dataset and four on the XSum dataset. Through our dataset, we aim to have a broad coverage of different types of errors made by neural summarization systems, with human judgements on their fine-grained factuality errors. Annotation Data For the annotation, we include model summaries from CNN/DM and XSum datasets as they present different characteristics. CNN/DM summaries are longer, with three sentences on average, while XSum has only single sentence summaries. Having longer summaries is crucial to identify discourse level errors. On the other hand, XSum summaries are more abstractive and include more factual errors on average (Maynez et al., 2020). For a diverse set of model summaries, we collect publicly available model outputs from different summarization models with differing factuality capabilities. For the CNN/DM dataset, we use model outputs from a LSTM Seq-to-Seq model (S2S) (Rush et al., 2015), a Pointer-Generator Network (PGN) model (See et al., 2017), a Bottom-Up Summarization (BUS) model (Gehrmann et al., 2018), a Bert based Extractive-Abstractive model (BertSum) (Liu and Lapata, 2019) and a jointly pretrained transformer based encoder-decoder model BART (Lewis et al., 2020). For the XSum dataset, we collect model outputs from a Topic-Aware CNN Model (Narayan et al., 2018), a Pointer-Generator Network (PGN) model, a randomly initialized (TransS2S) (Vaswani et al., 2017) and one initialized with Bert-Base (BertS2S) (Devlin et al., 2019). 2 Details of the models used are provided in A.1. Annotation Collection Using the above model generated summaries, we collect human annotations from three independent annotators for 250 articles from each dataset (with a total of 1250 model outputs on CNN/DM and 1000 on XSum). We annotate each sentence of a summary to break the judgement of factuality into smaller units. We present sentences in the context of the entire summary to identify discourse errors spanning multiple sentences. Annotations are a two step process: for each sentence in the summary, the annotator first selects whether the sentence is factual, and if marked not factual, identifies the category of each 2 As we use publicly available model outputs, the summaries across different datasets are from different models owing to their availability. error based on our typology. 3 A sentence can be annotated with more than one category of errors to account for multiple errors within a sentence. We conduct the annotation task on the Amazon Mechanical Turk (MTurk) platform. To achieve high quality crowd-sourced annotations, we build an intuitive interface 4 which combines: 1. Clear Instructions: We explain the annotation scheme without assuming linguistic knowledge and give several examples for each category. 2. Training and Evaluation: We setup training tutorials for first time users to train and provide feedback on the task. We also setup a qualification test which tests their understanding of our annotation scheme and require annotators to obtain >85% score to qualify. Further, we continuously evaluate annotators during the task against artificially generated factual errors to ensure continued high quality. 3. Fair Pay and Bonus: All workers are paid 50% more than the average American minimum wage. We offer bonuses for scores of 60% or above on the continuous evaluation, and for completing sets of 10 annotations. Further details on our interface are added in A.6 Inter-Annotator Agreement: We report inter-annotator agreement in terms of Fleiss Kappa  (Fleiss, 1971). Following Durmus et al. (2020), we report the percentage p of annotators that agree with the majority class. Each datapoint in our dataset corresponds to a sentence in a summary. We compute agreement on all 4942 annotated sentences. On the annotation of whether a sentence is factual or not we obtain  = 0 . 58 , with p = 91% of annotators agreeing with the majority class. As a comparison, Durmus et al. (2020) reports p = 76 . 7% average agreement. When all three annotators agree that a sentence is not factual, we obtain  = 0 . 39 with p = 73 . 9% of annotators agreeing with the majority class on the eight category annotation (seven categories of errors and other) which indicate a moderate agreement. 3 We experimented with Likert scale evaluation of full summaries in a pilot study. Such an annotation would not provide precise information about where in the summary an error appears and also resulted in lower agreement. Hence, we opted for sentence level judgements. 4 We make the interface available for future human annotations that follow our typology 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 BART BERTSum BUS PGN S2S CNN/DM 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 BERTS2S PGN TConvS2S TranS2S XSum PredE: Predicate EntE: Entity CircE: Circumstance CorefE: Coreference: LinkE: Connector OutE: Not in article GramE: Grammar OthE: Other Figure 2: Proportion of summaries with factual errors based on collected annotations, with breakdown of the categories of errors within. We find a Kohen Kappa of  = 0 . 86 indicating nearly perfect agreement. Previous work found agreement of  = 0 . 65 between three crowd annotators and expert annotations of factuality (Falke et al., 2019). Even with more than nine workers, they report agreement with expert annotations of at most  = 0 . 74 . This improvement validates the robustness of our annotation interface and protocol which achieves higher agreement with fewer workers. We evaluate the performance of different summarization models in terms of factuality. Figure 2 visualizes the percentage of summaries with factual errors for each category model and dataset, with a breakdown of proportion of different error types within each. A summary is considered incorrect if it contains at least one sentence with a factual error. A sentence contains a factual error if the majority of annotators indicate the presence of an error (here we do not consider annotations where all three annotators disagree on the category). How factual are generated summaries across different datasets? From our annotations, we observe that 60% of the summaries that were annotated contain at least one factual error. From Figure 2, we see that the XSum dataset has more factually incorrect model summaries (92%) than CNN/DM (43%). It poses more significant challenges in terms of factuality as all models produce > 80% summaries with factual errors, with the best model (BertS2S) producing 83% wrong summaries. On the CNN/DM dataset, while state-of-the-art pretrained models like BERTSum and BART have better factuality numbers, the percentage of factually incorrect summaries is still high (23% for BERTSum and 27% for BART). The proportion of errors across different categories vary widely between the two datasets. For the CNN/DM dataset, the most frequent classes of errors are Entity Error (EntE) and Coreference Error (CorefE). For the XSum dataset they are Out of Article Error (OutE) and Entity Error (EntE). Note that there are no discourse errors (CorefE, LinkE) in the XSum dataset because the data only contains single sentence summaries. Additionally, we observe that OthE makes up a very small percentage (  1%) of errors overall showing that our typology is complete with most errors being mapped to one of our existing categories. How factual are generated summaries across different models? From Figure 2, we observe that LSTM based models like S2S and BUS generate many incorrect summaries. Interestingly, PGN on CNN/DM has fewer summaries with factual errors (26%) compared to S2S (74%) and BUS (62%) potentially due to the extractive nature of CNN/DM and the copy based objective in PGN. PGN has been previously shown to produce highly extractive summaries on CNN/DM copying large portions of text (often entire sentences) (Gehrmann et al., 2018; Balachandran et al., 2021). On the more abstractive dataset XSum, PGN produces > 96% factually incorrect summaries. We also observe that large-scale pretrained models improve factuality on both datasets, as also noted by Durmus et al. (2020), with more significant gains on CNN/DM. On CNN/DM, BERTSum and BART display half the error rate of BUS. In contrast, on XSum, BertS2S improves over non-pretrained models by  10% only, showing that XSum poses a significant challenge for factuality even in pretrained models. models have higher proportion of Grammatical Errors (GramE) while transformer and CNN based models have a lower proportion. For pretrained transformer models, we observe that the improved error-rate on the CNN/DM dataset can be attributed to improvements at the frame level (PredE, EntE, CircE) while the discourse level errors still remain a challenge. Errors CorefE, LinkE account for a higher proportion of errors in BERTSum and BART compared to the other models. We propose the FRANK dataset resulting from the human annotation study as a common benchmark to assess different factuality metrics. We provide an evaluation protocol of factuality metrics, which controls for dataset biases, and a fine grained analysis of the strengths of each metric. The FRANK benchmark provides a diverse dataset for evaluating various metrics on their ability to capture factual errors. Notably, our benchmark has factual error diversity , as it covers all types of errors described in the typology in 2, and data diversity as it combines 2250 summaries from different systems and datasets. Our annotations go beyond binary labels of factuality on a summary by providing fine-grained category annotations for every sentence. This allows us to determine how well each metric can capture each type of error. Furthermore, through averaging of sentence level judgements, we can also obtain a factuality scores (0 to 1 range) for a summary. To measure the degree that automated metrics capture a certain characteristic, we compute their correlation with human judgements and report Pearson correlation and Spearman rank correlation along with their p-values. We evaluate different classes of metrics against the FRANK benchmark. We select four general summarization metrics. ROUGE, BLEU, and Meteor are n-gram based metrics and computed with respect to the reference summary. BERTScore (Zhang et al., 2020) computes BERT (Devlin et al., 2019) contextual embeddings on summary and source article and measures distances between matched embeddings. We select five metrics focused on factuality. As Goodrich et al. (2019), we use a simple OpenIE (Banko et al., 2007) baseline. This involves extracting OpenIE triples and matching them through sentence embeddings (Reimers 0.0 0.5 1.0 0.1 0.2 0.3 0.4 FEQA 0.0 0.5 1.0 0.2 0.4 0.6 0.8 FactCC BART BERTSum BUS PGN S2S BERTS2S-XS PGN-XS TConvS2S-XS TranS2S-XS All data 0.0 0.2 0.4 0.6 0.8 1.0 Human judgement of factuality 0.0 0.2 0.4 0.6 0.8 1.0 M e t r i c s c o r e Figure 3: Correlation between metrics and human judgement on subsets of data. The x and y axis represent the human judgement the metric scores respectively. The red line is a linear regression fitted on full data. Each dotted line is a linear regression fitted on a model-dataset subset. Each colored point has coordinates equal to average factuality judgement, and metric score for its corresponding partition. and Gurevych, 2019). FactCC (Kryscinski et al., 2020) and DAE (Goyal and Durrett, 2020) are entailment based metrics. FactCC operates with sentences as claims, while DAE uses dependency level entailment. FEQA (Durmus et al., 2020) and QAGS (Wang et al., 2020) are two question answering and generation metrics (QGA). More details on the differences between these metrics is in A.2. Since our benchmark contains diverse summaries from different datasets and models, dataset biases can hamper accurate reporting. In Figure 3, we visually show correlations between two factuality metrics (FEQA and FactCC) and human judgement on the entire data and on partitions of the data. For both metrics, we notice that the slope (an unscaled measure of correlation) of the line fitted through the entire data (red line) is significantly larger. In FEQA, the dotted lines (fitted on subsets of the data of each model and dataset) are almost horizontal. This likely indicates the presence of a confounding variable associated with the properties of each system and dataset. This can lead to false measures of high correlation if not accounted for. To address this, we suggest to control for confounding variables using partial correlations. We include details on partial correlations in the Appendix. In this case, both the system and the dataset are taken to be confounding variables. In Table 2, we report the partial Pearson correlation and Spearman rank correlation coefficients with human judgements for each metric, along with their How do different metrics correlate with human judgements? From Table 2 we observe that all metrics exhibit low correlations with human judgements of factuality. The best metric overall is FactCC with 0.20 Pearson and 0.30 Spearman correlation. Interestingly, we observe that general summarization metrics BLEU, Rouge, and METEOR, and the OpenIE baseline have statistically significant correlations with factuality, close to FactCC (  = 0 . 14 for Rouge-1 and METEOR versus  = 0 . 20 for FactCC). The entailment metrics (FactCC and DAE) have the two highest correlations and are statistically significant. The two QGA metrics have lower overall correlation. FEQA's correlation is not statistically significant. QAGS has low, but significant correlation of  = 0 . 06 . different datasets? In Figure 4, we observe that entailment metrics have significantly higher partial Pearson correlation on the CNN/DM dataset than XSum where their correlation is reduced by a factor of four. QAGS and the OpenIE baseline have similar behavior. This suggests that these metrics capture the error types from CNN/DM better that those from XSum. Specifically, XSum has uniquely high Out of Article (OutE) errors which they might not capture well. This also highlights the importance of data diversity in building and benchmarking factuality metrics to avoid overfit-ting to certain types of errors. How well do different metrics capture errors from pretrained and non-pretrained models? On the CNN/DM dataset we observe that entailment metrics and QAGS perform significantly better on non-pretrained models. This indicates that the artificial factual errors on which entailment metrics are trained on are closest to the mistakes that non-pretrained models make. This also suggests that the errors made by pretrained models might be more difficult to capture by these metrics. These trends are less clear on the XSum dataset which we again attribute to high Out of Article (OutE) errors in the pretrained and non-pretrained models (ref Figure 2) 5.4 Error Analysis Figure 4 shows partial Pearson correlation on six subsets of the data. To understand capabilities of metrics across the broad categories of errors (se-mantic frame errors, discourse errors, and content verifiability errors) we perform an ablation study. For each category, we compute the variation in partial correlation with errors from that category omitted. In Figure 5, we visualize the influence of a given type of error using the variation for each metric and category. A higher positive bar indicates that the error type was a significant contributer to the overall correlation (or metric highly correlates with error) causing the correlation without it to 0.0 0.2 0.4 CNN/DM no pretr. General Summarization metrics Unsurprisingly, we observe that Rouge L is best correlated with content verifiability errors (which contains Out of Article Errors) as n-gram matches detect them. Rouge L has negative correlation with semantic frame errors and low correlation with discourse level errors indicating that n-gram matching fails to capture them. We observe that OpenIE is more correlated with semantic frame errors. The metric matches entities and verifies the predicate that relates them and hence is able to capture semantic frame errors. BertScore has low correlation overall, being more correlated with content verifiability errors and negatively correlated with discourse errors. QGA metrics Both QGA metrics have negative correlation with discourse errors suggesting that QGA metrics are not able to capture coreference errors or discourse link errors potentially due to the entity oriented questions in their training data. FEQA additionally is also negatively correlated with semantic frame errors and has low positive correlation with content verifiability errors. In contrast QAGS is best correlated with semantic frame errors. correlation of all metrics with discourse errors suggesting that entailment at the dependency level can help model discourse errors (CorefE and LinkE). FactCC is nearly uncorrelated in this category, indicating that artificially generated factual errors need to go beyond simple pronoun swaps to train models to capture discourse errors. FactCC had best overall partial correlation which can be attributed to FactCC being able to capture semantic frame and content verifiability errors well. Kryscinski et al. (2019) and Fabbri et al. (2020) find that standard n-gram based metrics have low correlation with human judgements of factuality. Motivated by this, several automated metrics falling in two paradigms were proposed to improve the evaluation of factuality. Entailment Classification Goodrich et al. (2019); Kryscinski et al. (2020); Maynez et al. (2020); Goyal and Durrett (2020) model factuality as entailment classification breaking down the summary into smaller units, such as sentences, which are verified against the original article. However, modeling factuality as a classification task requires supervision on factual and hallucinated data. FactCC (Kryscinski et al., 2020) is trained on the CNN/DM dataset augmented with four types of artificial mistakes as supervision. Question Generation and Answering (QGA) FEQA (Durmus et al., 2020) and QAGS (Wang et al., 2020) are two metrics which reduce factuality evaluation to question generation and answering. These methods use a question generation model to obtain questions from the output summary and a question answering model to answer them, separately using the article and the output summary. (2020) have collected annotations on the CNN/DM and XSum dataset respectively. In this work we cover both datasets to ensure greater data diversity. Other efforts (Kryscinski et al., 2020; Wang et al., 2020; Durmus et al., 2020) were smaller in scale Durmus et al. (2020) and Kryscinski et al. (2020) annotated 200 and 503 sentences while Wang et al. (2020) annotated 470 summaries (we collect judgements on 2250 summaries). Crucially, all previous efforts portray factuality as a binary label without variations in degree or type of factual errors. In this work we provide a linguistically grounded typology of factual errors which we use to collect FRANK, a dataset of human annotations of 2250 summaries covering both CNN/DM and XSum datasets. We use FRANK to assess the factuality of summarization systems and benchmark recently proposed factuality metrics highlighting the types of errors they can capture. With the FRANK benchmark we have started moving away from a summary-level binary understanding of factuality. We have collected crowd annotations using the Amazon Mechanical Turk platform. Workers were paid 50% more than the average American minimum wage and offered additional bonuses as an incentive to maintain high quality work. No information about the workers will be released and worker IDs will be anonymized. The authors are grateful to the anonymous reviewers for their feedback, and to Anjalie Field, Rishabh Joshi, Alissa Ostapenko, Dheeraj Ra-jagopal, Evangelia Spiliopoulou, Shuly Wintner, and the members of the Tsvetshop group for their invaluable feedback and support in various stages of the project. This material is based upon work supported by the DARPA CMO under Contract No. HR001120C0124, and in part by the National Science Foundation under Grants No. IIS2040926 and No. IIS2007960. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily state or reflect those of the United States Government or any agency thereof.
abstain abstain abstain abstain method result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain objective abstain abstain objective method objective result result result abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain other method abstain method abstain method abstain abstain abstain method abstain method abstain abstain method abstain abstain method abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other result abstain abstain abstain abstain abstain result method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain method method abstain abstain abstain method method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other other other other abstain abstain method abstain method objective abstain method abstain abstain other other other other other other