Modern neural language models can produce remarkably fluent and grammatical text. So much, in fact, that recent work by Clark et al. (2021) has reported that conventional crowdsourcing can no longer reliably distinguish between machine-authored (GPT-3) and human-authored writing. As errors in machine generations become ever subtler and harder to spot, it poses a new challenge to the research community for robust machine text evaluation. We propose a new framework called SCARECROW for scrutinizing machine text via crowd annotation. To support the broad range of real machine errors that can be identified by laypeo-ple, the ten error categories of SCARECROW  such as redundancy , commonsense errors , and incoherence are identified through several rounds of crowd annotation experiments without a predefined ontology. We then use SCARECROW to collect over 41k error spans in human-written and machine-generated paragraphs of English language news text. We isolate factors for detailed analysis, including parameter count, training data, and various decoding-time configurations. Our approach successfully quantifies measurable gaps between human authored text and generations from models of several sizes, including fourteen configurations of GPT-3. In addition, our analysis unveils new insights, with detailed rationales provided by laypeople, e.g., that the commonsense capabilities have been improving with larger models while math capabilities have not, and that the choices of simple decoding hyperparameters can make remarkable differences on the perceived quality of machine text. We release our training material, annotation toolkit and dataset at https: //yao-dou.github.io/scarecrow/ . Clark et al. (2021) demonstrated the challenges of human evaluation in the era of GPT-3 (Brown  Equal contribution According to the Financial Times, Apple's been talking to "a small group of contract manufacturers to explore making an electric vehicle," which would ostensibly be an autonomous car. All this does sound like the loose ends of Apple's CarPlay rollout: hiring 1,200 engineers for the iOS team, building the CarPlay-specific testing track, developing a Lincoln Navigator, then poaching Burberry's head of product design to lead the integration of software and hardware. WWDC 2015 We know what you're thinking: Another Monday? et al., 2020), as crowd workers are no longer able to reliably distinguish GPT-3's generations from human-written text. Or are they? In this paper, we propose a new framework for systematically scrutinizing machine text so that even crowd workers, despite the known challenges reported by recent literature, can successfully critique seemingly fluent generations. We not only quantify a measurable gap between ma-1 7250 ERROR TYPE DEFINITION EXAMPLE Language Errors Grammar and Usage Missing, extra, incorrect, or out of order words ...explaining how cats feel emoticons ... To achieve this, we develop SCARECROW , a methodology for eliciting categorical judgements of errors in machine-generated text from crowd workers. One goal in natural language generation (NLG) is to produce fluent outputs which can be read by laypeople. As such, we propose that important errors to address are those which are recognized by readers without NLP expertise. Our framework allows crowd workers to annotate problems in model outputs at the span level. A single such annotation is shown in Figure 1. To make this possible, we establish a categorization of shortcomings commonly found in machine generated text (Table 1). This error schema covers a broad scope of problems as identified by experts, but has been honed according to what is salient to non-expert readers through several pilot rounds of crowd annotation without a fixed label set. The result is a framework that is usable by everyday people with minimal training, but covers the error phenomena found in real machine-generated text. Labeling spans of text using specific error types creates a picture of contemporary model generations with an unprecedented level of detail. In contrast to judging text holistically (Celikyilmaz et al., 2021), insights from this method are specific and practical, as it measures exactly how and where problems arise. We conduct a large-scale analysis of human-written and machine-generated text using SCARECROW , collecting 13k annotations of 1.3k paragraphs, amassing 41k spans labeled with error type, severity, and an explanation. Through this, we characterize in which ways GPT-3's generations are better than those of previous models, and which aspects do not improve with increased data and parameters. We also provide a rigorous error analysis of text generated by several other contemporary language models, examining the impact of model size, training data, and decoding strategy. We provide our detailed annotator training system and task interface so that future researchers may employ and refine them for error analyses of machine-generated text. We hope this will contribute to the standardization of NLG human evaluation (Howcroft et al., 2020). We perform a large-scale annotation of errors in English news text generated by five sources (four models and ground truth articles). We present Figures 2, 3, and 4 as summaries of our main results. As a reminder to readers, Grover (Zellers et al., 2019) is the same model size and architecture as GPT-2 XL (Radford et al., 2019), but trained in-domain (on news text). As such, our results cover three increasing model sizes (GPT-2 Small, XL, and GPT-3 (Brown et al., 2020)), one change in domain (Grover), and ground-truth text (Human). For GPT-3, we also study a variety of decoding configurations (Figure 4). The main quantity we measure (on y -axes) is span coverage , which is the average portion of tokens that ends up covered by annotations of a particular error type. Since it is possible that multiple spans nest or overlap, there is no upper bound for this quantity. (See Figure 12 for a comparison of span coverage with other measurement alterna-tives.) Figure 2 measures span coverage for each type of span separately, Figure 3 stacks them, and Figure 4 removes non-error spans (reader issues) before adding them (as in Figure 3, but without showing the individual types). The following are our key findings. 1. Scaling pays off to improve Encyclopedic , Commonsense , and Incoherent errors (Fig. 2). These error categories decrease with in-domain training (Grover) and larger model size (GPT-3). The Needs Google and Technical Jargon span categories both have a humans highest trend, and both fall under reader issues : problems that are not necessarily errors , but that still prevent full comprehension 3 7252 GPT-2 S GPT-2 XL Grover-Mega GPT-3 Human 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 S pan c o v e r age Average Span Coverage Across Models Bad_Math Commonsense Encyclopedic Grammar_Usage Incoherent Needs_Google Off-prompt Redundant Self-contradiction Technical_Jargon Figure 3: Average portion of tokens covered by span annotations, broken down by error type. 2. Scaling benefits plateau for Off-Prompt , Bad Math , and Grammar and Usage errors (Fig. 2). These three error categories see a model plateau in error reduction when scaling to GPT-3. Of these error types, humans still commit fewer Off-Prompt (more: E.1) and Grammar and Usage errors, but Bad Math appears saturated for our domain. 3. Self-Contradiction and Redundant errors exhibit more complex scaling behavior (Fig. 2). We roughly categorize these trends as rising and falling : increasing for medium or large-scale models, but dropping for human-authored text. Text generated by GPT-2 Small is so often incoherent that there is little possibility for Self-Contradiction (more: E.2), and the increase in Redundant errors varies based on how errors are counted (more: E.3). 4. Human-authored text produces the most reader issues (Figs. 2 and 3). or factual verification of the text (more: E.4). Furthermore, human-authored text is not free from error annotations (Figure 3). This can serve either as a control for baseline error rates (more: E.6), or as a mechanism for critiquing human writing. impact (Figure 4). For the previous findings, we fix the sampling configuration for all models to an apples-to-apples setup for fair comparison: topp = 0.96, (softmax) temperature = 1, and no frequency penalty (i.e., word repetition penalty; defined precisely in 5.2, Equation 1). To study the effects of these decoding settings, we annotate text generated by GPT-3 using a variety of values for topp and temperature, both with and without a frequency penalty. To our surprise, the decoding hyperparameters considerably affected error rates (more: E.5). As seen in Figure 4, the worst sampling procedure for GPT-3 (argmax sampling with no frequency penalty) performed even worse than GPT-2 XL. But the best sampling procedure (surprisingly, also argmax sampling, but with a frequency penalty) produced text with as few apparent SCARECROW error spans as those authored by humans (more: E.6). All of these findings are discussed in more detail in Appendix E. 3 Evaluation of Natural Language Generation We make our study in the area of open-ended natural language generation, a loose term for generating longer texts with an increased level of creative freedom. The common factor in all open-ended generation tasks such as story, blog, and dialog generation is the wide and diverse nature of target outputs. Lexically and even semantically dissimilar responses to the same prompt could be equally valid. For example, a model prompted with the blog title Recipes for success this Holiday season could describe how to roast a turkey or strategies for dealing with the stresses of holiday travel. This allowable variation poses a particular difficulty for the evaluation of generation systems. Traditionally, text generation quality for tasks like machine translation or graph-to-text generation has been measured by word overlap with human-authored references (Papineni et al., 2002; Lin, 2004). Though measures like BLEU allow for multiple references, they break down when the space of allowable outputs is large, as in open-ended generation. Recently introduced metrics seek to remedy this problem (Hashimoto et al., 2019; Pillutla et al., 2021), but the gold standard for evaluating generated text is still human judgment. However, current approaches to eliciting human 4 7253 judgement of generated text often do not provide detailed insight into where models are making progress, where they are failing, and the scope of these failures. A/B-style testing allows for directly comparing one system against others (Clark and Smith, 2021), but can only express relative improvements. Simple Likert scale judgements can assess text quality, but do not explain why a generated text receives a given rating, or which segment of the text is problematic. Insights into model failures often come instead from a small scale expert analysis of outputs. However, these error analy-ses, once a staple of NLP research, have become less common in recent years, perhaps due to their small size and high variance. A hypothesis of the current work is that a well designed error analysis annotation framework could be used by crowdworkers to annotate large amounts of text, thereby providing detailed information about model progress and failures as well as actionable directions for future research. Such a framework would be easy to learn, reusable, and independent of particular models or experimental conditions. In what follows, we outline the details of such a method. This section describes the high-level annotation methodology for SCARECROW . Our annotations consider two segments of text: a one-sentence prompt, and a one-paragraph generation. The prompt is human-written. It provides both starting tokens for model generation, as well as context for humans to evaluate whether a model is able to stay on-promptboth topically and factually. Annotators know that the prompt is written by a human. The generation is either text sampled from a language model, or the human-authored continuation to the prompt. Annotators, who do not know whether the generation came from a model or humans, assess this text. A paragraph length (80145 tokens) is chosen to balance expressiveness with scope. For expressiveness, models must be given a sufficient number of tokens to express their capabilities lexically, syntactically, and semantically. One paragraph allows for significantly more variation than a single sentence. On the other hand, assessing multiple paragraphs is challenging, both Inconsistent about how many moons Mars has. as a crowdsourcing task itself, and because it broadens the kinds of errors to include larger narrative scope. We leave extensions of SCARECROW to longer narrative lengths for future work. Annotators select spans that contain problems in the generation. The spans are automatically snapped to word boundaries. We choose spans to balance specificity (i.e., vs. simply commenting on the text as a whole) with ease of use (vs. imposing a more structured annotation schema). We instruct workers to select the smallest span minimally a single wordthat contains an issue. Sometimes this involves an entire phrase, sentence, 5 7254 or multiple sentences. We aim for specificity because during aggregation, it is possible to back off annotations to larger spans, but not the inverse. Once they select a span, workers (1) label the error type, (2) choose a severity level, and (3) explain their reasoning behind the error. Workers use the annotation interface shown in Figure 5 to mark a span with these three steps. We describe each step in greater detail in the next three sections. Each selected span is labeled with exactly one error type. Multiple errors may be marked with partially or fully overlapping spans in the case that one text segment contains multiple problems. We chose ten error types to balance three criteria: linguistic analysis, observed errors in generated text, and capabilities of everyday people with one to two hours of training. 1 We developed the schema by starting with the first two criteria (lin-guistic analysis and observed errors), and refining it over several pilot annotation studies, with 30 crowd workers performing 750 total annotations of 60 paragraphs before beginning data collection. We broadly group the errors into three categories: language errors, factual errors, and reader issues. Language errors are issues with internal and external structure of text: which ideas are expressed, and whether they are expressed coherently and consistently. Factual errors denote that the information presented is known to be incorrect. Reader issues, on the other hand, are cases where the text is too technical or obscure to assess its factuality. Hence, reader issues are not errors, per se, but regions where a reader would need assistance outside of the text itself for comprehension. Errors naturally vary in how jarring they are to a reader. We define three error severity levels, and ask annotators to pick one for each error. The severity levels are as follows. (1) Almost no impact on quality; just a small problem. (2) Understandable, but difficult; what's written is still comprehensible, but there's clearly an issue. (3) Very difficult to understand; the error almost completely ruins the text. We provide examples of each severity in Appendix B.1. In this paper, we omit an analysis of the severity labels (except for an illustration in Figure 12), but include it in our data release for future work to explore. Finally, we ask annotators to explain their reasoning behind each error in natural language. We provide example explanations during training, but do not impose strict guidelines. This paper primarily focuses on quantitative error analysis, but we anticipate the error explanations may warrant future investigation. We use Amazon Mechanical Turk (AMT) for all data collection. Training We first pay each worker $40 to take an extensive qualification task, which both trains them in the span categorization scheme and quizzes their understanding. We pass workers if they score  90 points out of 100 points (details in Appendix B.2). Annotation Workers annotate each paragraph using a custom annotation interface (shown partially in Figure 5), for which we pay $3.50. We calculated $3.50 per annotation by aiming to pay workers at least $15/hour. After several annotation rounds, we observed considerable variation in time per annotation, 2 so this cost should not be necessarily seen as a requirement for SCARECROW annotations. We collect 13k human annotations of 1.3k paragraphs using SCARECROW , resulting in over 41k spans. We consider four model configurations to test recent state-of-the-art transformer-based (Vaswani et al., 2017) models. GPT-2 Small (Radford et al., 2019) The 117M parameter variant of GPT-2, which is pretrained on WebText, without additional fine-tuning. GPT-2 XL (Radford et al., 2019) The 1.5B parameter variant of GPT-2, (WebText, no fine-tuning). Grover-Mega (Zellers et al., 2019) The 1.5B parameter variant of Grover, a model with the same architecture and parameter count of GPT-2, trained on news articles and their metadata. GPT-3 DaVinci (Brown et al., 2020) The 175B parameter variant of GPT-3, which is trained on a version of the Common Crawl web scrape with additional filtering and deduplicating. We consider three main hyperparameters when sampling from models: p for top-p or nucleus sampling (Holtzman et al., 2020), an alternative to top-k ; 3 t for the softmax temperature ; and f.p. for frequency penalty . The frequency penalty scales a token's likelihood based on how many times it was already generated by applying the following modification to the model's output: (cid:96) i ( t )  (cid:96) i ( t )  c <i ( t )   f (1) where (cid:96) i ( t ) is the model's output for token t at the i -th position, 4 c <i ( t ) is the count of token t 's sampled occurrences prior to the i -th position, and  f is the frequency penalty. We omit studying presence penalty , another hyperparameter offered for GPT-3, simply due to annotation budget constraints. To compare models as consistently as possible, we set identical decoding strategies for our primary data collection. We refer to this as the apples-to-apples decoding setup throughout the paper: p = 0 . However, we also wish to study the effects of these decoding strategies. We annotate generations from the strongest available model (currently, GPT-3) varying the following parameters: 3 We omit separate studies of top-k , due to results presented by Holtzman et al. (2020), and OpenAI's removal of top-k from the GPT-3 API. 4 While (cid:96) i ( t ) is defined to be logits (un-normalized logprobabilities), because it is un-normalized, we anticipate that it is simply the model's output before the log( softmax (  )) is applied. See OpenAI's description of frequency and presence penalties: https://beta.openai.com/docs/ api-reference/parameter-details p  { 0 . 4 , 0 . 7 , 0 . 9 , 0 . 96 } t  { 0 . 0 (argmax) , 0 . 4 , 0 . 7 , 1 . 0 } f.p. { 0 (none) , 1 (full) } For budget reasons, we only vary p and t independentlyi.e., we set p = 0 . 96 when varying t , and t = 1 . 0 when varying p . We use news articles as the sources of prompts for models to condition on for generation. Specifically, we use news articles found in the Common Crawl. We select the first sentence as the prompt. Our use of news text is constrained by two factors. First GPT-3 is trained on the Common Crawl, from 2016 through 2019. We wish to avoid testing GPT-3 by generating from articles it saw during training, due to the possibility of copying (Carlini et al., 2021). Second, news articles began heavily covering the COVID-19 pandemic beginning around February 2020. Though testing models' capabilities to generate text about unseen events is a valuable line of study, the distribution shift caused by COVID-19 in news writing about all aspects of life is difficult to overstate. As such, to make the comparison more amenable to models' training data, we consider news articles from January 2020. We select articles where there is a known topicsuch as Food or Sports from the Common Crawl metadata, to allow for studying any effect of coarse-grained subject. We generate between 80 and 145 tokens 5 from each model as a continuation to the first sentence of the news article. We stop generating when we heuristically detect the first sentence boundary after 80 tokens. If the model does not end a sentence between 80 and 145 tokens, we sample again. For the Human setting, we use the remainder of the article, similarly stopping after the first sentence boundary after 80 tokens. Crowdsourcing Workers first complete training and qualification tasks. We provide more details in 4.7. From pilot studies, we discovered that each error, depending on its severity and clarity, has only a 5 Counted by Stanza tokenization (Qi et al., 2020), not byte-pair encoding (BPE) or whitespace-separated tokens. low to moderate chance of being identified by each worker. However, most worker-identified errors were truly problems. In other words, annotators labeled issues with high precision and low recall. To account for this, we have 10 workers annotate each paragraph. We examine the agreement and variability of annotations in Appendix C. Dataset statistics We provide detailed dataset statistics in Appendix D. 6 Error Prediction A natural question is: using this data, can machines learn to detect and classify errors in machine generated text? Task We frame this problem as a span classification task. Given a span from a generated text, the goal is to classify its error type or output No Error if there is none. Positive examples for each error class are taken from our data. We sample random spans that were not labeled with any error type as negative examples. To ensure a breadth of span lengths, we sample 3 negative spans for every length of error span in the generated text. We split the generated texts into train, development, and test sets using 1063 texts (28029 error spans), 100 texts (2538 spans) and 100 texts (2677 spans) respectively. Model We use a standard span classification model inspired by Wadden et al. (2019). This model encodes every generated text using a pretrained language model (RoBERTa-large). Spans are represented with the final layer of this encoding. Following previous work, we concatenate the start and end tokens with a task-specific learned length embedding. The resulting vector is passed through a feedforward network which reduces its dimensionally to the number of error categories plus a No Error option. The resulting model has 357M trainable parameters. The model is trained to minimize the cross entropy of the correct span category. We train for 15 epochs using AdamW with a learning rate of 10  6 . We validate after each epoch and use the checkpoint with the lowest validation loss (epoch 8). Evaluation To evaluate the error prediction model, we use per-token precision, recall, and F 1 score per error category. We classify every span up to length 30 in a generated text. We take as gold labels the aggregated human error spans collected Error Model Human P R F 1 P R F 1 Bad Math  0  0.72 0.14 0.24 Commonsense 0.77 0.06 0.10 0.17 0.02 0.04 Encyclopedic  0  0.22 0.03 0.05 Grammar and Usage 0.29 0.23 0.26 0.30 0.04 0.08 Incoherent 0.59 0.34 0.43 0.69 0.15 0.24 Off-Prompt 0.67 0.29 0.41 0.88 0.31 0.46 Redundant 0.23 0.82 0.36 0.88 0.35 0.50 Self-Contradiction 0.08 0.23 0.12 0.51 0.09 0.16 Technical Jargon 0.18 0.74 0.29 0.61 0.12 0.20 Needs Google 0.59 0.96 0.73 0.78 0.20 0.32 Table 2: Model prediction results against combined spans of 10 annotators, compared with humans scored as one-vs-rest (i.e., 1-vs-9). in our data. In other words, models predict the combined spans of all 10 annotators. For comparison, we also report as Human the average metrics of one annotator versus the others (i.e., 1-vs-9). 6 Results Table 2 shows the error prediction capability of this model in terms of precision and recall. As we noted earlier, a single human annotator can be thought of as a high precision, low recall judge. These results bear out this claim. For all but one category, humans have higher precision annotations. However, the models trained on the aggregation of human labels can achieve considerably higher recall. For half of the error categories, this leads to higher model F 1 scores than the human annotators. We see that the model is successful at identifying information that human's would have to manually verify ( Needs Google ), achieving nearly perfect recall with precision close to 0.6. The model can also identify Grammar and Usage , Incoherent , and Redundant errors with higher recall than an individual human annotator, though at the cost of precision (sometimes in the .20s). Automated evaluation metrics such as BLEU (Pa-pineni et al., 2002), ROUGE (Lin, 2004), METEOR (Banerjee and Lavie, 2005), and BERTScore 6 The difference in available references (10 for models, 9 for humans) mean this setup makes it easier for models to score higher in precision, and for humans to score higher in recall. Despite this, humans still achieve higher precision, and models still achieve higher recall. (Zhang et al., 2019) compute a generation's score based on a (set of) reference(s). Their use is well-established in tasks like machine translation and summarization, but they are less helpful in open-ended text generation, where there is a vast diversity of possible high-quality continuations. Recent studies propose automated metrics for open-ended text generation evaluation such as: Perception Score (Gu et al., 2021), which diffuses evaluation onto a multidimensional space and assigns a single score; UNION (Guan and Huang, 2020), which learns to distinguish human-written stories from negative samples by generating perturbations of human-written stories; and MAUVE (Pillutla et al., 2021), which compares the distribution of machine-generated text to that of human language. An alternate recent approach to assessing open-ended text generation was presented in TuringAd-vice (Zellers et al., 2021), where crowd workers assess machine-generated advice in response to Reddit posts. In their error analysis, Zellers et al. connect problems in generated text to core NLP tasks, such as Self-Contradiction errors as instances of failed natural language inference (Monz and de Rijke, 2001), or Off-Prompt errors as cases of failed reading comprehension (Richardson et al., 2013). While past work has attempted to guide text generation using discriminative models trained for such tasks (Holtzman et al., 2018), it remains an open challenge. Comparative human evaluations of natural language generations ask annotators to rank system outputs relative to each other. Text is typically evaluated using a few global criteria, such as fluency and relevance, using discrete (e.g., 5-point) (Sai et al., 2020) or continuous scales (Novikova et al., 2018). Recent work even automates this approach, running a human evaluation alongside automatic metrics on leaderboard submissions (Khashabi et al., 2021). In the RoFT system (Dugan et al., 2020), annotators attempt to detect the boundary between humanand machine-written text as a proxy for assessing quality. Table 3 summarizes the differences between these schemes and SCARECROW . See Celikyilmaz et al. (2021) for a recent survey of text generation evaluation techniques across both human and automatic metrics. While these approaches may be helpful sometimes (Card et al., 2020)at ranking systems, they do not give us insight into exactly which parts of a generation fall short, and why . One approach Method GC SET DE RR EE RS SA Likert-Scale (cid:88) (cid:88) (cid:88) RankME (cid:88) (cid:88) (cid:88) RoFT (cid:88) (cid:88) (cid:88) SCARECROW (cid:88) (cid:88) (cid:88) (cid:88) Table 3: Comparison of different natural language generation human evaluations. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, 9 7258 Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam Mc-Candlish, Alec Radford, Ilya Sutskever, and Dario Amodei. related to or annotation method is pursued by Wood et al. (2018), who develop a collaborative mobile app where users draw graffiti commentary on news articles. SCARECROW aims to assess model generations the way we would critique human-written text: by locating, coarsely categorizing, and explaining problems. We present SCARECROW , a method for identifying and explaining issues in generated text. Along with the annotation framework, we present an analysis of the SCARECROW method applied to several large neural language models in an open-ended news generation task. We release our data and methodology to the community. The authors thank members of xlab for their feedback on this work. This research is supported in part by NSF (IIS-1714566), DARPA MCS program through NIWC Pacific (N66001-19-2-4031), DARPA SemaFor program, and Allen Institute for AI.
abstain abstain abstain objective abstain method abstain abstain objective abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain method abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain method method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain other method method abstain method abstain method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain method abstain abstain abstain abstain abstain method abstain abstain method method method method method abstain abstain method abstain abstain abstain method method method method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other other other other other other other other other other other other abstain other other other objective method method abstain other other