Document-level information extraction (IE) tasks have recently begun to be revisited in earnest using the end-to-end neural network techniques that have been successful on their sentence-level IE counterparts. Evaluation of the approaches, however, has been limited in a number of dimensions. In particular, the precision/recall/F1 scores typically reported provide few insights on the range of errors the models make. We build on the work of Kummerfeld and Klein (2013) to propose a transformation-based framework for automating error analysis in document-level event and (N-ary) relation extraction. We employ our framework to compare two state-of-the-art document-level template-filling approaches on datasets from three domains; and then, to gauge progress in IE since its inception 30 years ago, vs. four systems from the MUC-4 (1992) evaluation. 1 1 Introduction Although information extraction (IE) research has almost uniformly focused on sentence-level relation and event extraction (Grishman, 2019), the earliest research in the area formulated the task at the document level . Consider, for example, the first large-scale (for the time) evaluations of IE systems  e.g. MUC-3 (1991) and MUC-4 (1992). Each involved a complex document-level event extraction task: there were 24 types of events, over a dozen event arguments (or roles ) to be identified for each event; documents could contain zero to tens of events, and extracting argument entities (or role fillers ) required noun phrase coreference resolution to ensure interpretability for the end-user (e.g. to ensure that multiple distinct mentions of the * These authors contributed equally to this work. 1 Our code for the error analysis tool and its output on different model predictions are available at https://github. com/IceJinx33/auto-err-template-fill/. same entity in the output were not misinterpreted as references to distinct entities). The task was challenging: information relevant for a single event could be scattered across the document or repeated in multiple places; relevant information might need to be shared across multiple events; information regarding different events could be intermingled. In Figure 1, for example, the DISEASE "Newcastle" is mentioned well before its associated event is mentioned (via the triggering phrase "the disease has killed"); that same mention of "Newcastle" must again be recognized as the DISEASE in a second event; and the COUNTRY of the first event ("Honduras") appears only in the sentence describing the second event. In fact, the problem of document-level information extraction has only recently begun to be revisited (Quirk and Poon, 2017; Jain et al., 2020; Du et al., 2021b,a; Li et al., 2021; Du, 2021; Yang et al., 2021) in part in an attempt to test the power of end-to-end neural network techniques that have been so successful on their sentence-level counterparts. 2 Evaluation, however, has been limited in a number of ways. First, despite the relative complexity of the task, approaches are only evaluated with respect to their overall performance scores (e.g. precision, recall, and F1). Even though scores at the role level are sometimes included, no systematic analysis or characterization of the types of errors that occur is typically done. The latter is needed to determine strategies to improve performance, to obtain more informative cross-system and cross-genre comparisons, and to identify and track broader advances in the field as the underlying approaches evolve. To date, for example, there has been no attempt to directly compare the error landscape and distribution of 2 See, for example, Zhang et al. (2019), Du and Cardie (2020) and Lin et al. (2020) for within-sentence event extraction; Akbik et al. (2018), and Akbik et al. (2019) for named entity recognition (NER); and Zhang et al. (2018) and Luan et al. (2019) for sentence-level relation extraction. newly developed neural IE methods with that of the largely hand-crafted systems of the 1990s. In this work, we first introduce a framework for automating error analysis for document-level event and relation extraction, casting both as instances of a general role-filling, or template-filling task (Juraf-sky and Martin, 2021). Our approach converts predicted system outputs into their gold standard counterparts through a series of template-level transformations (Figure 2) and then maps combinations of transformations into a collection of IE-based error types. Examples of errors include duplicates, missing and spurious role fillers, missing and spurious templates, and incorrect role and template assignments for fillers. (See Figure 3 for the full set). Next, we employ the error analysis framework in a comparison of two state-of-the-art document-level neural template-filling approaches, DyGIE++ (Wadden et al., 2019) and GTT (Du et al., 2021b), across three template-filling datasets (SciREX, ProMED (Patwardhan and Riloff, 2009) 3 , and MUC-4). Finally, in an attempt to gauge progress in the information extraction field over the past 30 years, we employ the framework to compare the performance of four of the original MUC-4 systems with the two newer deep-learning approaches to document-level IE. 4 We find that (1) the best of the early IE models  which strikes a better balance between 3 http://www.promedmail.org 4 The 1992 model outputs are available in the MUC-4 dataset released by NIST, available at https: //www-nlpir.nist.gov/related_projects/muc/muc_data/muc_data_index.html. precision and recall  outperforms modern models that exhibit much higher precision and much lower recall; (2) the modern neural models make more mistakes on scientific vs. news-oriented texts, and missing role fillers is universally the largest source of errors; and (3) modern models have clear advantages over the early IE systems in terms of accurate span extraction, while the early systems make fewer mistakes assigning role fillers to their roles. Aside from the original MUC-4 evaluation scoring reports (Chinchor, 1991), which included counts of missing and spurious role filler errors, there have been very few attempts at understanding the types of errors made by IE systems and grounding those errors linguistically. Valls-Vargas et al. (2017) proposed a framework for studying how different errors propagate through an IE system; however, the framework can only be used for pipelined systems, not end-to-end ones. On the other hand, automated error analysis with linguistically motivated error types has been used in other sub-fields of NLP such as machine-translation (Vilar et al., 2006; Zhou et al., 2008; Farrs et al., 2010; Kholy and Habash, 2011; Ze-man et al., 2011; Popovic and Ney, 2011), coreference resolution (Uryupina, 2008; Kummerfeld and Klein, 2013; Martschat and Strube, 2014; Martschat et al., 2015) and parsing (Kummerfeld et al., 2012). Recently, generalized automated error analysis frameworks involving human-in-the-loop testing like Errudite (Wu et al., 2019), CHECK 3961 LIST (Ribeiro et al., 2020), CrossCheck (Arendt et al., 2021), and AllenNLP Interpret (Wallace et al., 2019) have successfully been applied to tasks like machine comprehension and relation extraction (Alt et al., 2020). Closest to our work are Kummerfeld et al. (2012) and Kummerfeld and Klein (2013), which use model-agnostic transformation-based mapping approaches to automatically obtain error information in the predicted structured output. As in Jurafsky and Martin (2021), we will refer to document-level information extraction tasks as template-filling tasks and use the term going for-ward to refer to both event extraction and document-level relation extraction tasks. Given a document, D , and an IE template specification consisting of a predetermined list of roles R 1 , R 2 , ..., R i associated with each type of relevant event for the task of interest, the goal for template filling is to extract from D , one output template, T for every relevant event/relation e 1 , e 2 , . . . , e n present in the document. Notably, in the general case, n  0 and is not specified as part of the input. In each output template, its roles are filled with the corresponding role filler(s), which can be inferred or extracted from the document depending on the predetermined role types. We consider two role types here: 5 Set-fill roles , which must be filled with exactly one role filler from a finite set supplied in the template specification. An example of a set-fill role in Figure 1 is STATUS , which can be confirmed , possible , or suspected . String-fill roles , whose role filler(s) are spans extracted from the document, or left empty if no corresponding role filler is found in the document. VICTIMS , DISEASE and COUNTRY are string-fill roles in Figure 1. Some string-fill roles allow multiple fillers; for example, there might be more than one VICTIMS . Importantly, for document-level template filling, exactly one string should be included for each role filler entity (typically a canonical mention of the entity), i.e. coreferent mentions of the same entity are not permitted. Evaluation. We use the standard (exact-match) F1 score (Chinchor, 1991) to evaluate the output 5 There are potentially more role types depending on the dataset (e.g. normalized dates, times, locations); we will not consider those here. produced by a template-filling system: F 1 = 2  Precision  Recall Precision + Recall 4 Methodology: Automatic Transformations for Error Analysis Similar to the work of Kummerfeld and Klein (2013), our error analysis approach is system-agnostic, i.e. it only uses system output and does not consider intermediate system decisions. This allows for error analysis and comparison across different kinds of systems  end-to-end or pipeline; neural or pattern-based. Given inputs consisting of the system-predicted templates and gold standard templates (i.e. desired output) for every document in the target dataset, our error analysis tool operates in three steps. For each document, 1. Perform an optimized mapping of the associated predicted templates and gold templates. 2. Apply a pre-defined set of transformations to convert each system-predicted template into the desired gold template, keeping track of the transformations applied. The first stage of the error analysis tool involves matching each system-predicted template to the best-matching gold template for each document in the dataset. In particular, the overall F1 score for a given document can vary based on how a predicted template is individually matched with a gold template (or left unmatched). Specifically, for each document, we recursively generate all possible template matchings  where each predicted template is matched (if possible) to a gold template. In particular, for a document with P predicted templates and G gold templates, the total number of possible template matchings is: 1 + (cid:32) P 1 (cid:33) G + (cid:32) P 2 (cid:33) G ( G  1) + ... + G ! ( G  P )! , if G  P  0 1 + (cid:32) P 1 (cid:33) G + (cid:32) P 2 (cid:33) G ( G  1) + ... + (cid:32) P G (cid:33) G ! , if G  P < 0 = min ( P,G ) (cid:88) i =0 (cid:32) P i (cid:33) G ! ( G  i )! Note that template matching can result in unmatched predicted templates ( Spurious Templates ), as well as unmatched gold templates ( Missing Templates ). Next, for each predicted-gold pair in a template matching, we iterate through all its roles and recursively generate all possible mention matchings , in each of which a predicted role filler is matched (if possible) to a set of coreferent gold role fillers. Similar to template matching, the process of mention matching can also result in unmatched predicted role fillers ( Spurious Role Fillers ) and unmatched coreferent sets of gold role fillers ( Missing Role Fillers ). Through the process, each predicted role filler increases the denominator of the total precision by 1, and each set of coreferent gold role fillers increases the denominator of total recall by 1. Whenever there is a matched mention pair in which the predicted role filler has an exact match to an element of the set of coreferent gold role fillers, this adds 1 to the numerator of both precision and recall. These counts are calculated for each template matching. Using precision and recall, the total F1 score across all the slots/roles is calculated and the template matching with the highest total F1 score is chosen. If there are ties, the template matching with the fewest errors is chosen (see Section 4.3). The second part of the error analysis tool involves changing the predicted templates to the desired gold templates with the help of a fixed set of transformations as detailed below. Alter Span transforms a role filler into the gold role filler with the lowest span comparison score ( SCS ). The tool provides two options for computing the SCS between two spans, and each depends only on the starting and ending indices of the spans. 6 SCS can be interpreted as distance and is 0 between two identical spans, and 1 for non-overlapping spans. The two modes are given as follows: a) absolute : This mode captures the (posi-tive) distance between the starting (and ending) character offsets of spans x and y in the document, and scales that value by the sum of the lengths of x and y , capping it at a maximum of 1. SCS = max (cid:16) 1 , | x start  y start | + | x end  y end | length ( x )+ length ( y ) (cid:17) This mode captures the degree of disjointedness between spans x and y by dividing the length of the overlap between the two spans with respect to each of their lengths, multiplying those two fractions, and subtracting the final result from 1. If si is the length of the intersection of x and y , and neither x nor y have length 0, SCS is calculated as shown below; otherwise, SCS is 1. overlap = min ( x end , y end )  max ( x start , y start SCS is between 0 and 1 (not inclusive), and if there is no overlap between the spans, the SCS is 1. The order of comparison of the spans doesn't change the SCS score for both modes. As the absolute mode is less sensitive to changes in span indices as compared to the geometric mean, we chose geometric mean for our analysis, as tiny changes in index positions result in a bigger change in the SCS score. another role within the same template. Remove Duplicate Role Filler removes a role filler that is coreferent to an already matched role filler. Remove Cross Template Spurious Role Filler removes a role filler that would be correct if present in another template (in the same role). Remove Spurious Role Filler removes a role filler that has not been mentioned in any of the gold templates for a given document. Introduce Role Filler introduces a role filler that was not present in the predicted template but was required to be present in the matching gold template. Remove Template removes a predicted template that could not be matched to any gold template for a given document. Introduce Template introduces a template that can be matched to an unmatched gold template for a given document. For a given document, all singleton Alter Span and Alter Role transformations, as well as sets of Alter Span + Alter Role transformations, are applied first. The other transformations are applied in the order in which they were detected, which is dependent on the order of predicted and gold template pairs in the optimized matching and the order of the slots/roles in the template. The transformations in Section 4.2 are mapped onto a set of IE-specific error types as shown in Figure 3. In some cases, a single transformation maps onto a single error, while in others a sequence of transformations is associated with a single error. Full details are in Appendix A. 5 Document-level IE Datasets Our experiments employ three document-level information extraction datasets. We briefly describe each below. Dataset statistics are summarized in Table 1. MUC-4 (MUC-4, 1992) consists of newswire describing terrorist incidents in Latin America provided by the FBIS (Federal Broadcast Information Services). We converted the optional templates to required templates and removed the subtypes of the incidents as done in previous work (Chambers, 2013; Du et al., 2021b) so that the dataset is transformed into standardized templates. The roles chosen from the MUC-4 dataset are PERPIND (individual perpetrator), PERPORG (organization perpetrator), TARGET (physical target), VICTIM (human target), and WEAPON which are all string-fill roles, as well as INCIDENT TYPE which is a set-fill role with six possible role fillers: attack , kidnapping , bombing , arson , robbery , and forced work stoppage . As seen in Table 1, 44.59% of the documents have no templates, which makes the identification of relevant vs. irrelevant documents critical to the success of any IE model for this dataset. ProMED 8 (Patwardhan and Riloff, 2009) consists of just 125 annotated tuning examples and 120 annotated test examples, describing global disease outbreaks by subject matter experts from ProMED. We use the tuning data as training data and reserve 10% of the test data, i.e. 12 examples, to create a de-velopment/validation set. 19.83% of the documents in the dataset have no templates. The roles that we extract from the dataset are STATUS , COUNTRY , DISEASE , and VICTIMS . DISEASE , VICTIMS , and COUNTRY are string-fill roles 9 ; STATUS is a set-fill role with confirmed , possible , and suspected as the possible role filler options. SciREX (Jain et al., 2020) consists of annotated computer science articles from Papers with Code 10 . We focus specifically on its 4-ary relation extraction subtask. The roles present in each relation are MATERIAL (DATASET ), METRIC , TASK , and METHOD which are all string-fills. We convert the dataset from its original format to templates for our models, and remove individual role fillers (entities) that have no mentions in the text. 11 We also remove any duplicate templates. 12 During preprocessing, we remove malformed words longer than 25 characters, as the majority of these consist of concatenated words that are not present in the corresponding text. In our experiments, we train and test two neural-based IE models, described briefly below, on the MUC-4, ProMED, and SciREX datasets. Note that 8 http://www.promedmail.org 9 In the ProMED dataset, COUNTRY is a set-fill role, but since countries are explicitly mentioned in most of the documents, we can treat this role as a string-fill. 10 https://paperswithcode.com 11 According to Jain et. al., around 50% of relations in the SciREX dataset contain one or more role fillers that do not appear in the corresponding text. These relations are removed during evaluation for our end-to-end task. https://github.com/allenai/SciREX/blob/master/README.md 12 Removing unmentioned entities sometimes eliminates differences between templates. This results in some templates becoming identical or making some templates contain information that is a subset of the information present in another template. Thus, we only keep one of these processed templates. to create the training data for both the DyGIE++ and GTT models, we use the first mention of each role filler in the document as the mention to be extracted. DyGIE++ with Clustering We use DyGIE++  a span-based, sentence-level extraction model  to identify role fillers in the document and associate them with certain role types. During training, the maximum span length enumerated by the model is set to 8 tokens as in Wadden et al. (2019) for the SciREX dataset and 11 tokens for the ProMED dataset. We use bert-base-cased and al-lenai/scibert_scivocab_uncased for the base BERT and SciBERT models respectively, which both have a maximum input sequence length of 512 tokens. To aggregate entities detected by DyGIE++ into templates, we use a clustering algorithm. For the SciREX dataset, we adopt a heuristic approach that assumes there is only one template per document, and in that template, we assign the named entities predicted by DyGIE++ for a document to the predicted role types. For the ProMED dataset, we use a different clustering heuristic that ensures that each template has exactly one role filler for the COUNTRY and DISEASE roles, as detailed in the dataset annotation guidelines. Also, since STATUS has the value confirmed in the majority of the templates, every template predicted has its STATUS assigned as confirmed . GTT is an end-to-end document-level template-generating model. For the MUC-4 and SciREX datasets, GTT is run for 20 epochs, while for ProMED it is run for 36 epochs, to adjust for the smaller size of the dataset. All other hyperparameters are set as in Du et al. (2021b). We use the same BERT and SciBERT base models as described in the DyGIE++ architecture above, both with a maximum input sequence length of 512 tokens. We first discuss the results of DyGIE++ and GTT on SciREX, ProMED, and MUC-4; and then examine the performance of these newer neural models on the 1992 MUC-4 dataset vs. a few of the best-performing IE systems at the time. Table 2 shows the results of evaluating DyGIE++ and GTT on the SciREX, ProMED, and MUC-4 datasets. We can see that all models perform substantially worse on scientific texts (ProMED, SciREX) as compared to news (MUC-4) , likely because the model base is pretrained for general-purpose NLP applications (BERT) or there are not enough examples of scientific-style text in the pretraining corpus (SciBERT). In addition, models seem to perform better on the news-style ProMED dataset than the scientific-paper-based long-text SciREX dataset. This can be explained by the fact that all four models handle a maximum of 512 tokens as inputs, while the average length of a SciREX document is 5401 tokens. Thus, a majority of the text is truncated and, hence, unavailable to the models. Nevertheless, we see an increase in F1 scores for all SciBERT-based models when compared to their BERT counterparts for the SciREX dataset. The same trend is seen for DyGIE++ for ProMED, but not for GTT. This can be explained by the fact that GTT (SciBERT) has more Missing Template errors than GTT (BERT). So even if GTT (SciBERT) performs better on the scientific slot VICTIMS , i.e. it extracts more scientific information, it does not identify relevant events as well as GTT (BERT), reducing the F1 score across the remaining slots. From the error count results in Figure 4, we see that GTT makes fewer Missing Template errors than DyGIE++ on the MUC-4 dataset (86 vs. 97). However, there is no significant difference 3966 SciREX ProMED MUC-4 DyGIE++ (BERT) 22.47% 35.01% 45.79% DyGIE++ (SciBERT) 25.39% 38.15% GTT (BERT) 21.54% 44.64 % 49.00% GTT (SciBERT) 27.68 % 42.96% Table 2: F1 Scores for the Neural Models on SciREX, ProMED, and MUC-4 Precision Recall F1 GE NLToolset 56.69% 52.09% 54.29% NYU PROTEUS 34.23% 31.28% 32.69% SRI FASTUS 48.47% 38.42% 42.86% UMass CIRCUS 48.62% 39.04% 43.30% GTT (BERT) 63.18 % 40.02% 49.00% DyGIE++ (BERT) 61.90% 36.33% 45.79% Table 3: Precision, Recall, and F1 scores for models on the MUC-4 dataset. in the number of missing templates between the two models on the ProMED and SciREX datasets. This could be because DyGIE++ is prone to over-generation  there are significantly more Spurious Role Filler and Spurious Template errors as compared to the results of GTT. Since we use heuristics that create templates based on the extracted role fillers, this increases the probability that there was a possible match to a gold template, reducing the number of Missing Template Errors. We can also conclude that DyGIE++ is worse at coreference resolution when compared to GTT as DyGIE++ makes more Duplicate Role Filler errors across all datasets. Overall, we find that the major source of error for both GTT and DyGIE++ across all the datasets is missing recall in the form of Missing Role Filler and Missing Template errors. Table 3 presents the precision, recall, and F1 performance on the MUC-4 dataset for early models from 1992 alongside those of the more recent DyGIE++ and GTT models. We summarize key findings below. The best of the early models (GE NLToolset) performs better than either of the modern models. It does so by doing a better job balancing precision and recall, whereas GTT and DyGIE++ exhibit much higher precision and much lower recall. The early models have more span errors than the modern DyGIE++ and GTT models. The representative kinds of span errors from the 1992 model outputs are shown in Table 4. One interesting difference between the span errors in the early models and the modern models is that the predicted mentions include longer spans with more information than is indicated in the best gold mention match. Some could be due to errors in dataset annotation; for example, maoist shining path group versus shining path but a significant number of the span errors occur as the early models seem to extract the entire sentence or clause which contains the desired role filler mention. The modern models tend to leave off parts of the desired spans, and if they do predict larger spans than required, are only off by a few words. The early models have fewer Missing Template and Missing Role Filler errors as compared to the modern models. However, the former also have more Spurious Template and Spurious Role Filler errors than the latter, indicating these models mitigate the issue of Missing Templates through over-generation. The early models have fewer Incorrect Role errors as compared to modern models. However, since all the models make relatively few such errors, it suggests that role classification for predicted mentions is not a major problem for modern models. The main source of error for both early and modern models is missing recall due to missing templates and missing role fillers. This strongly suggests future systems can maximize their performance by being less conservative in 3967 Error Counts for Models on the MUC 4 Dataset M o d e l s DyGIE++ (BERT) GTT (BERT) GE NYU SRI UMass Number of Errors 0 250 500 750 1000 Span Error Duplicate Role Filler Duplicate ParLally Matched Role Filler Spurious Role Filler Missing Role Filler Incorrect Role Incorrect Role + ParLally Matched Filler Wrong Template Role Filler Wrong Template For ParLally Matched Role Filler Wrong Template + Wrong Role Wrong Template + Wrong Role + ParLally Matched Filler Spurious Template Spurious Template Role Filler Missing Template Missing Template Role Filler W i t h i n T e m p l a t e W i t h i n + C r o ss T e m p l a t e T e m p l a t e D e t e c t i o n Span Error Duplicate Role Filler Duplicate Partially Matched Role Filler Spurious Role Filler Missing Role Filler Incorrect Role Incorrect Role + Partially Matched Filler Wrong Template Role Filler Wrong Template For Partially Matched Role Filler Wrong Template + Wrong Role Wrong Template + Wrong Role + Partially Matched Filler Spurious Template Spurious Template Role Filler Missing Template Missing Template Role Filler 1 Figure 4: Automated Error Analysis Results (Error Counts) for Models on the MUC-4 dataset. 9 Conclusion As new models for information extraction continue to be developed, we find that their predicted error types contain insights regarding their shortcomings. Analyzing error patterns within model predictions in a more fine-grained manner beyond scores provided by commonly used metrics is important for the progress of the field. We introduce a framework for the automatic categorization of model prediction errors for document-level IE tasks. We used the tool to analyze the errors of two state-of-the-art models on three datasets from varying domains and compared the error profiles of these models to four of the earliest systems in the field on a dataset from that era. We find that state-of-the-art models, when compared to the earlier manual feature-based models, perform better at span extraction but worse at template detection and role assignment. With a better balance between precision and recall, the best early model outperforms the relatively high-precision, low-recall modern models. Missing role fillers remain the main source of errors, and scientific corpora are the most difficult for all systems, suggesting that improvements in these areas should be a priority for future system development. This work explores subtypes of Spurious Role Filler errors extensively, however, we would like to further analyze Missing Role Filler and template-level errors for more fine-grained error subtypes and the linguistic reasons behind why they occur. Due to the pairwise comparisons between all predicted and gold mentions in a role for all pairs of predicted and gold templates in an example, the error analysis tool is slow when the number of both the predicted and gold templates as well as the number of role fillers in the templates is high. Thus, we would also like to improve the time complexity of our template (and mention) matching algorithms using an approach like bipartite matching (Yang et al., 2021). Currently, the error analysis tool reports exact match precision/recall/F1 which is more suitable for string-fill roles. We would like to extend the tool to further analyze set-fill roles by implementing metrics such as false-positive rate. We used a limited number of models in this paper as we aimed to develop and test the usability of our error analysis tool. In the future, however, we would like to test our tool on a wider range of models, in addition to running more experiments in order to reach more generalizable conclusions about the behavior of IE models. We thank the anonymous reviewers and Ellen Riloff for their helpful comments(!) and Sienna Hu for converting the 1992 model outputs to a format compatible with our error analysis tool. Our research was supported, in part, by NSF CISE Grant 1815455 and the Cornell CS Department CSURP grants for undergraduate research.
abstain abstain abstain abstain method abstain abstain other abstain abstain other abstain abstain abstain abstain other abstain abstain method abstain abstain method objective other other other other other other abstain method other other other method other other other other other method method other method other other abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain method abstain method abstain abstain method abstain method method method method other other abstain method other abstain abstain abstain method abstain method method method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain result abstain abstain abstain result other abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other