Recent advances in prompt-based learning have shown strong results on few-shot text classification by using cloze-style templates. Similar attempts have been made on named entity recognition (NER) which manually design templates to predict entity types for every text span in a sentence. However, such methods may suffer from error propagation induced by entity span detection, high cost due to enumeration of all possible text spans, and omission of inter-dependencies among token labels in a sentence. Here we present a simple demonstration-based learning method for NER, which lets the input be prefaced by task demonstrations for in-context learning. We perform a systematic study on demonstration strategy regarding what to include (entity examples, with or without surrounding context), how to select the examples, and what templates to use. Results on in-domain learning and domain adaptation show that the model's performance in low-resource settings can be largely improved with a suitable demonstration strategy (e.g., 4-17% improvement on 25 train instances). We also find that good demonstration can save many labeled examples and consistency in demonstration contributes to better performance. 1 1 Introduction Neural sequence models have become the de facto approach for named entity recognition (NER) and have achieved state-of-the-art results on various NER benchmarks (Lample et al., 2016; Ma and Hovy, 2016; Liu et al., 2018). However, these data-hungry models often rely on large amounts of labeled data manually annotated by human experts, which are expensive and slow to collect (Huang et al., 2020; Ding et al., 2021b), especially for specialized domains ( e.g. , research papers). To  Authors contributed equally. improve NER performance on low-resource (label scarcity) settings, prior works seek auxiliary supervisions, such as entity dictionary (Peng et al., 2019; Shang et al., 2018; Yang et al., 2018; Liu et al., 2019) and labeling rules (Safranchik et al., 2020; Jiang et al., 2020), to either augment human-labeled data with pseudo-labeled data, or incorporate meta information such as explanation (Lin et al., 2020; Lee et al., 2020, 2021), context (Wang et al., 2021), and prompts (Ding et al., 2021a; Cui et al., 2021) to facilitate training. However, such methods have the following challenges: (1) human efforts to create auxiliary supervisions (e.g., dictionaries, rules, and explanations); (2) high computational cost to make predictions. For example, Ding et al. (2021a) shows effectiveness on entity type prediction given the entity span by constructing a prompt with the structure [entity span] is [MASK]" . However, when the entity span is not given, cloze-style prompts need to be constructed over all the entity candidates in the sentence with the structure [entity candidate] is [MASK]" to make a prediction (Cui et al., 2021). Such brute-force enumerations are often expensive. In this paper, we propose demonstration-based learning (Gao et al., 2021; Liu et al., 2021), a simple-yet-effective way to incorporate automatically constructed auxiliary supervision. The idea was originally proposed in prompt-based learning to show some task examples before the cloze-style template so that the model can better understand and predict the masked slot (Gao et al., 2021). This paper proposes modified version of demonstration-based learning for NER task. Instead of reformatting the NER task into the cloze-style template, we augment the original input instances by appending automatically created task demonstrations and feed them into pre-trained language models (PTLMs) so that the model can output improved token representations by better understandings of the tasks. Unlike existing efforts which require additional hu-2687 man labor to create such auxiliary supervisions, our model can be automatically constructed by picking up proper task examples from the train data. Moreover, unlike approaches that need to change the format of token classification into cloze-style mask-filling prediction which can neglect latent relationships among token labels, our approach can be applied to existing token classification module in a plug-and-play manner (See Figure 1 (a) vs (b)). We investigate the effectiveness of task demonstration in two different low-resource settings: (1) in-domain setting which is a standard NER benchmark settings where the train and test dataset come from the same domain; and (2) domain-adaptation setting which uses sufficient labeled data in source domain to solve new tasks in a target domain. Here, we study which variants of task demonstration are useful to train an accurate and label-efficient NER model and further explore ways to adapt the source model to target domain with a small amount of target data. We propose two ways of automatic task demonstration construction: (1) entity-oriented demonstration selects an entity example per entity type from train data to construct the demonstration. It allows the model to get a better sense of entity type by showing its entity example; and (2) instance-oriented demonstration retrieves instance example similar to input sentence in train data. It allows the model to get a better sense of the task by showing similar instances and their entities. We show extensive experimental results on CoNLL03, Ontonotes 5.0 (generic domain), and BC5CDR (biomedical domain) over 3 different templates and 5 selection/retrieval strategies for task demonstrations. For entity-oriented demonstration , we present 3 selection strategies to choose appropriate entity example per entity type: (1) random randomly selects entity example per entity type; (2) popular selects the entity example which occurs the most per entity type in the train data; and (3) search selects the entity example per entity type that shows the best performance in the development set. And for instance-oriented demonstration , we present 2 retrieval strategies to choose appropriate instance example ( SBERT (Reimers and Gurevych, 2019) vs. BERTScore (Zhang et al., 2020)). Our findings include: (1) good demonstration can save many labeled examples to reach a similar level of performance in low-resource settings. Our approach consistently outperforms standard fine-tuning by up to 3 points in terms of F1 score (p-value < 0.02); (2) demonstration becomes more effective when we also provide context. For example, not only showing Fischler is PER', but also the sentence that contains Fischler' as person, such as France backed Fischler's proposal'; and (3) consistency in demonstration contributes to better performance. Our experiments show that using consistent demonstration for all instances rather than varying per instance lead to better performance 2 Related Works NER with additional supervision Recent attempts addressing label scarcity have explored various types of human-curated resources as auxiliary supervision. One of the research lines to exploit such auxiliary supervision is distant-supervised learning. These methods use entity dictionaries (Peng et al., 2019; Shang et al., 2018; Yang et al., 2018; Liu et al., 2019) or labeling rules (Safranchik et al., 2020; Jiang et al., 2020) to generate noisy-labeled data for learning a NER model. Although these approaches largely reduce human efforts in annotation, the cross-entropy loss may make the model be overfitted to the wrongly labeled tokens due to noisy labels (Meng et al., 2021). Another line of research is incorporating such auxiliary supervision during training and inference in a setting of supervised learning. These approaches usually 2688 incorporate external information that is encoded including POS labels, syntactic constituents, dependency relations (Nie et al., 2020; Tian et al., 2020), explanations (Lin et al., 2020; Lee et al., 2020, 2021), retrieved context (Wang et al., 2021) and prompts (Ding et al., 2021a; Cui et al., 2021). Demonstration-based Learning Providing a few training examples in a natural language prompt has been widely explored in autoregressive LMs (Brown et al., 2020; Zhao et al., 2021). Such prompt augmentation is called demonstration-based learning (Gao et al., 2021). This is designed to let prompt be prefaced by a few examples before it predicts label words for [MASK] in the cloze-style question. Recent works on this research line explore a good selection of training examples (Gao et al., 2021) and permutation of them as demonstration (Kumar and Talukdar, 2021). In this section, we introduce basic concepts of named entity recognition, standard fine-tuning for sequence labeling, and domain adaptation for sequence labeling. We then formally introduce our goal  generating task demonstration and then developing a learning framework that uses them to improve NER models. Here, we let x = [ x (1) , x (2) , . . . x ( n ) ] denote the sentence composed of a sequence of n words and y = [ y (1) , y (2) , . . . y ( n ) ] denote the sequence of NER tags. The task is to predict the entity tag y ( i )  Y for each word x ( i ) , where Y is a pre-defined set of tags such as {B-PER, I-PER, . . . , O}. In standard fine-tuning , NER model M parameterized by  is trained to minimize the cross entropy loss over token representations h = [ h (1) , h (2) , . . . h ( n ) ] which are generated from the pre-trained contextualized embedder as follows: L =  n (cid:88) i =1 log f i,y i ( h ;  ) (1) where f is the model's predicted conditional probability that can be either from linear or CRF layers. We let D train and D test denote the labeled train and test dataset, respectively, consisting of { ( x i , y i ) } . Here, we expect the number of labeled instances in D train is extremely limited (e.g., N < 50 ). Given such small labeled instances, our goal is to train an accurate NER model with task demonstrations compared to standard fine-tuning and show the effectiveness of demonstration-based learning. We evaluate the trained models on D test . Domain adaptation aims to exploit the abundant data of well-studied source domains to improve the performance in target domains of interest. We consider two different settings: (1) label-sharing setting in which the label space L = (cid:8) l 1 , . . . , l | L | (cid:9) (e.g., l i = P ERSON ) of source-domain data S and target-domain data T are equal; (2) labeldifferent setting which L is different. In domain adaptation, we first train a model M s on source-domain data S . Next, we initialize the weights of the new model M t by weights of M s . Here, we can either transfer the whole model weights or only the weights of contextualized embedder from M s to M t . Then, we further tune M t on target-domain data T . In our preliminary experiments, we find that transferring only the embedder from M s to M t is much more effective than transferring the whole model weights (See first rows in Table 2 and Table 3). For this paper, we focus on the effectiveness of our models to adapt to the target domain with a T , for which the number of instances is extremely limited. We then 2689 In this work, we focus on how to create effective task demonstration  x to elicit better token representations for x , and then we propose an efficient learning framework that can be improved by the effect of [ x ;  x ] . This section introduces the concepts of demonstration-based learning , and provides details of the approach. Here, we study example sampling strategies and templates to construct the demonstration (Sec 4.1) and how we can train the NER model with the demonstration (Sec 4.2). Task demonstration  x = [ [SEP] ;  x 1 ;    ;  x l ] is constructed by selecting entity example e or retrieving instance example s from D train ( T train for domain adaptation) and modifying by template T to form  x i . The demonstration sequence  x is then appended to the original input x to create a demonstration-augmented input [ x ;  x ] . Here, [SEP] in front of  x is to separate x and  x . The key challenge of constructing task demonstration is to choose appropriate e or s and template T that can be helpful to demonstrate how the model should solve the task. As shown in Figure 2, we categorize the demonstration into (1) entity-oriented demonstration ; and (2) instance-oriented demonstration by whether we choose e or s respectively, for demonstration. Entity-oriented demonstration. Given an entity type label set L = (cid:8) l 1 , . . . , l | L | (cid:9) , we select an entity example e per label l from D train . Then, we modify it using template T . To select e per each l , we first enumerate all the e  D train and create a mapping { l i : [ e 1 , . . . , e n ] | l i  L } between l and corresponding list of entities. Then for each label l , we select e by three selection strategies: (1) random randomly chooses e from the list; (2) popular chooses e that occurs the most frequently in the list; and (3) search conducts grid search over possible entity candidates per label. Here, we sample top-k frequent entities per label, and search over combinations of entity candidates ( = k | L | ). We find the best combination that maximizes the F1 score on the dev set D dev . Here,  x i for every x i is different in random while  x i for every x i is same in popular and search . Instance-oriented demonstration. Given an input x , we retrieve an instance example s that is the most relevant to the input from D train . Then, we modify the s along with its { e, l }  s by template T . For retrieval, we present two strategies: (1) SBERT (Reimers and Gurevych, 2019) retrieves semantically similar sentence using pre-trained bi-encoder. It produces CLS embeddings independently for an input x and s  D train , and compute the cosine similarity between them to rank s  D train ; (2) BERTScore (Zhang et al., 2020), which is originally used as a text generation metric, retrieves token-level semantically similar sentence by computing a sum of cosine similarity between token representations of two sentences. Since the NER task aims to token classification, sentence-level similarity may retrieve a sentence that is semantically relevant but has no relevant entities. Fixed vs Variable demonstration. As described in previous sections, the demonstration in some strategies varies per instance while in others it stays fixed globally. We can divide the demonstration strategies into two categories: (1) Variable demonstration: random , SBERT , BERTScore (2) Fixed demonstration: popular , search Demonstration template. (1) no-context shows selected e per l with a simple template  e is l .", without including the spe-2690 cific sentence where the entities show up. Between each pair of ( e, l ) (of different entity labels l ), we concatenate with separator [SEP] . This template is only applied to the entity-oriented demonstration. (2) context in entity-oriented demonstration shows selected e per l along with an instance sentence s that contains e as a type of l . For each triple of ( e, l, s ) , it is modified into  s . e is l ." and concatenated with [SEP] . For instance-oriented demonstration, it shows the retrieved instance s along with all the entities mentioned in the sentence e  s . It is modified into  s . e 1 is l 1 . . . . e n is l n .". (3) lexical in entity-oriented demonstration also shows selected e per l along with an instance sentence s . But here we only show s , which the entity span e is replaced by its label string l . For instance-oriented demonstration, we show retrieved s by replacing e  s with the corresponding l . We expect such templates can form labeling rules and let the model know how to label the sentence. Transformer-based standard fine-tuning for NER first feeds the input sentence x into a transfomer-based PTLMs to get the token representations h . The token representations h are fed into a CRF layer to get the conditional probability p  ( y | h ) , and the model is trained by minimizing the conditional probability by cross entropy loss: L =  n (cid:88) i =1 log p  ( y | h ) (2) In our approach, we define a neural network parameterized by  that learns from a concatenated input [ x ;  x ] . For both model training and inference, we feed the input and retrieve the representations: [ h ;  h ] = [ h (1) ,...h ( n ) ,  h (1) ,...  h ( n ) ] = embed([ x ;  x ]) (3) As shown in Figure 1, we then feed h into the CRF layer to get predictions and train by minimizing the conditional probability p  ( y | h ) as Equation 2. For domain adaptation, we first train M s with standard fine-tuning. Then, transfer the weights of embedder of M s to M t and further fine-tune M t with our approach. We consider three NER datasets as target tasks. We consider two datasets for a general domain Dataset Label Train Data 25 50 CoNLL03 PER (Person) 16.0  3.52 29.2  4.52 LOC (Location) 15.6  3.92 30.4  4.07 ORG (Organization) 21.8  2.31 32.6  3.77 MISC (Miscellaneous) 11.0  2.52 15.6  2.33 Ontonotes 5.0 PER (Person) 10.8  2.22 21.4  4.02 LOC (Location) 16.0  3.52 25.0  7.32 ORG (Organization) 13.8  3.48 24.2  6.17 MISC (Miscellaneous) 23.8  5.56 62.6  7.93 BC5CDR Disease 25.8  6.01 29.2  4.52 Chemical 51.0  7.49 65.8  7.12 Table 1: Data statistics. ( CoNLL03 (Tjong Kim Sang, 2002), Ontonotes 5.0 (Weischedel et al., 2013)) and one dataset for a bio-medical domain ( BC5CDR (Li et al., 2016)). CoNLL03 is a general domain NER dataset that has 22K sentences containing four types of general named entities: LOCATION , PERSON , ORGANIZATION , and MISCELLANEOUS entities that do not belong in any of the three categories. Ontonotes 5.0 is a corpus that has roughly 1.7M words along with integrated annotations of multiple layers of syntactic, semantic, and discourse in the text. Named entities in this corpus were tagged with a set of general 18 well-defined proper named entity types. We split the data following (Pradhan et al., 2013). BC5CDR has 1,500 articles containing 15,935 CHEMICAL and 12,852 DISEASE mentions. To show its effectiveness in few-shot NER, we also show baselines of few-shot NER methods NNShot and StructShot (Yang and Katiyar, 2020). NNshot is simple token-level nearest neighbor classification system while StructShot extends NNshot with a decoding process using abstract tag transition distribution. Here, both the classification model and the transition distribution should be pre-trained on the source dataset. Thus, we consider this as domain adaptation setting. We implement all the baselines and our frameworks using PyTorch (Paszke et al., 2019) and Hugging-Face (Wolf et al., 2020). We set the batch size and learning rate to 4 and 2e-5, respectively, and use bert-base-cased model for all the experiments. For each variant, we run 50 epochs over 5 different sub-samples and 3 random seeds with early-stopping 20 and show its average and stan-2691 Demonstration / Method Strategy Template CoNLL03 Ontonotes 5.0 BC5CDR 25 50 25 50 25 50 BERT+CRF w/o demonstration -52.72  2.44 62.75  0.98 38.97  4.62 54.51  3.27 52.56  0.46 60.20  2.01 BERT+CRF w/ SBERT lexical 48.92  2.81 57.68  0.37 36.58  4.61 44.47  2.58 49.41  0.94 51.98  2.14 Instance-oriented demonstration ( variable ) context 53.62  1.64 64.21  1.87 42.18  5.21 53.07  3.46 54.71  2.09 59.78  1.47 BERTScore lexical 49.55  3.18 58.85  1.06 35.42  3.88 44.70  2.41 49.37  0.19 51.61  2.45 ( variable ) context 53.97  1.52 64.66  2.04 37.56  5.29 53.13  3.22 54.81  2.11 59.63  1.94 BERT+CRF w/ random no-context 53.95  1.89 63.31  2.14 42.25  3.61 55.71  3.82 53.58  0.48 59.97  1.89 Entity-oriented demonstration ( variable ) lexical 55.20  2.24 63.60  2.32 44.02  4.73 56.31  3.83 53.79  0.61 59.65  1.71 context 54.84  2.12 63.51  2.83 43.57  3.73 56.76  3.69 54.08  0.97 59.94  1.70 popular no-context 54.34  3.33 64.30  2.76 43.02  4.33 56.65  3.35 53.86  0.86 60.51  1.77 ( fixed ) lexical 56.22  3.88 64.95  2.04 45.31  5.02 58.24  3.17 54.14  0.67 60.67  1.58 context 56.52  3.34 64.47  2.35 45.52  4.69 58.40  3.24 54.31  0.80 61.31  1.51 search no-context 54.63  2.12 64.50  2.76 42.88  5.41 56.96  4.09 53.97  1.32 60.84  2.14 ( fixed ) lexical 56.57  3.61 65.11  2.71 44.87  5.09 58.51  3.42 54.39  1.57 60.76  2.12 context 57.00  4.03 64.82  3.16 45.74  5.57 59.00  3.27 55.83  1.25 62.87  2.41 Table 2: In-domain performance comparison (F1-score) on CoNLL03, Ontonotes 5.0, and BC5CDR by different number of training instances. dard deviation of F1 scores. Unlike existing sampling methods for few-shot NER (Yang and Kati-yar, 2020), in which the training sample refers to one entity span in a sentence, we consider a real-world setting that humans annotate a sentence. We sub-sample data-points by random sampling with a constraint that sampled instances should cover all the BIOES labels (Chiu and Nichols, 2016) in the whole dataset. For Ontonotes, we aggregate all other entity types rather than person, location, and organization into miscellaneous to set the label sharing setting for domain adaptation experiments. Table 1 presents statistics of average number of entities per entity type over 5 different sub-samples. We first compare the overall performance of all baseline models and our proposed framework with the amount of training data 25 and 50 to show the impact of our approach in a low-resource scenario, assuming a task that needs to be annotated from scratch. Then, we show performance analysis to show the effectiveness of our approach and whether the model really learns from the demonstration. In-domain setting In Table 2, we can observe that most variants of demonstration-based learning consistently and significantly (with p-value < 0.02) outperform the baseline by a margin ranging from 1.5 to 7 F1 score in three low-resource NER datasets (25, 50 train instances respectively). It demonstrates the potential of our approach for serving as a plug-and-play method for NER models. Domain adaptation setting First, we observe that simple domain adaptation technique can improve the performance (First rows of Table 2 vs. Table 3). Here, we only transfer the embedder weights of M s to M t , and we expect the performance gain can be attributed to the embedder of M s , which is trained in task adaptive pre-training manner on NER task formats (Gururangan et al., 2020). In Table 3, we can see that the most variants of demonstration-based learning allow the source model M s to be adapted to the target domain in fast with a small amount of target data T , compared to baselines without demonstration including few-shot NER methods. Entity vs. Instance-oriented demonstration. instance-oriented demonstration performs worse than entity-oriented demonstration due to the diffi-culty of finding an appropriate similar instance in a low resource train data. In our analysis, we find that the average cosine similarity between retrieved example s and input x is less than 0.4 which shows many of the retrieved examples are not appropriate similar examples to the input. Fixed vs. Variable demonstration. As mentioned in section 4.1, random doesn't pick a fixed set of demonstrations the same way as popular and search . Instead, it picks random demonstrations for each input instance. In a low-resource setting, there are often no significantly popular entities. Therefore, the fact that popular outperforms random in our experiments might suggest that the consistency of demonstration selection, rather than popularity of selected entities, is a crucial factor in better few-shot learning. To test this, we randomly select one entity per entity type and attach it as the demonstration to all instances, we call it ( fixed random ). As shown in Figure 4, it outperforms random and is on par with popular and search . We believe this serves as evidence for two hypotheses: (1) consistency of demonstration is essential to performance, and (2) in low-resource settings, the effectiveness of combinations of entities as demonstrations might be a rather random function and not too affected by the combination's collective popularity in the training dataset, which further implies that the idea of search is on the right track. Performance in other model variants To show the effectiveness of demonstration-based learning as plug-and-play method, we present performance in other model variants: bert-large-cased , LM Strategy Template In-domain LabelSharing CoNLL03 CoNLL03->Ontonotes 25 50 25 50 BL -52.08  2.02 66.42  2.14 63.50  0.96 70.59  1.16 RB-59.67  4.65 70.17  3.93 68.43  2.09 74.11  1.19 RL -59.15  2.93 71.51  3.44 68.16  2.65 74.45  1.02 BL popular context 57.60  3.37 67.11  2.31 64.09  2.95 70.88  1.09 RB popular context 59.76  4.27 70.21  3.41 69.09  2.63 74.53  1.32 RL popular context 59.99  2.16 72.15  3.81 68.78  2.89 74.93  1.07 Table 4: Performance comparison (F1-score) with various backbone LMs: bert-large-cased (BL) ; roberta-base (RB) ; and roberta-large (RL) . roberta-base and roberta-large . As shown in Table 4, our method shows consistent improvement over baselines (p-value < 0.05). It shows that demonstration-based learning can be applied to any other model variants and output better contextualized representations for NER tasks and show its potential for scalability. Effectiveness of search . search consistently outperforms all other strategies. It shows that not only the entity selection, but also the combination of entity examples per each entity type affects the performance. To see whether it consistently outperforms the baseline over various low-resource data points, we show the performance trend of entity-oriented demonstration in Figure 5. Templates of entity-oriented demonstration. entity-oriented demonstration becomes more effective when not only showing the entity example per each entity type, but also the corresponding instance example as a context. context and lexical consistently outperform no-context . We explore other templates as well, and these three are the best among them. We present details on Appendix A. To see whether the order of entity type in entity-oriented demonstration affects the performance, we present analysis of entity type permutation, e.g., person organization location miscellaneous . There is no 2693 25 # of train instances (CoNLL03) 50 52 54 56 58 60 F 1 S c o r e No demonstration Permutation (context) # of train data: 25 with/without the demonstration (denoted by O" and X", respectively) at training and inference time. clear pattern of which entity type order is better (spearman correlation between F1-scores over different entity type orders with 25 and 50 training instances < 0), but all the permutations outperform the baseline as shown in Figure 6, which show that demonstration-based learning can be effective regardless of the order (See Appendix Figure 8). Demonstration perturbation. To investigate whether the model really learns from demonstration, we explore the performance of our approach with perturbed demonstration which selects random entities, labels, and context sentences as demonstration. Here, we present two studies: (1) Test perturbation which train with correct demonstration and test with perturbed demonstration; and (2) Train-test perturbation which both train and test with perturbed demonstration. Figure 7 shows perturbed demonstration disturbs the model in a large margin for both case. This shows that the model affects by demonstration, and proper demonstration can improve the model's performance. Full results are available in Appendix Table 9. Effects of demonstration in train & inference. Table 5 shows the effects of demonstration in training and inference stage. A comparison of row 0 with row 3 shows that applying demonstration in the training stage but not in the inference stage would make the model perform worse than the fine-tuning baseline. This is another evidence that CoNLL BC5CDR Ontonotes 25 Train Instances 35 40 45 50 55 60 65 F 1 S c o r e Train-test Perturbation (context) Original Perturbation Fully supervised setting. Table 6 shows the performance in fully supervised setting, where the train data is sufficient. We can see that demonstration-based learning yields similar performance as baselines (p-value < 0.1), which shows that demonstrations are rather redundant when data is abundant. In this paper, we propose demonstration-based learning for named entity recognition. Specifically, we present entity-oriented demonstration and instance-oriented demonstration and show that they successfully guide the model towards better understandings of the task in low-resource settings. We observe that entity-oriented demonstration is more effective than instance-oriented demonstration , and search strategy consistently outperforms all other variants. Moreover, we find that consistent demonstration for all the instances is crucial to the superior performance of our approach. We believe that our work provides valuable cost reduction when domain-expert annotations are too expensive and opens up possibilities for future work in automatic demonstration search for few-shot named entity recognition.
abstain abstain abstain method method abstain objective abstain abstain abstain other abstain other abstain abstain abstain objective objective abstain method result abstain abstain objective objective objective abstain abstain result result method abstain result abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other method abstain abstain method abstain method abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain abstain method method method abstain abstain abstain abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain method abstain method other abstain abstain method method abstain abstain abstain abstain method abstain method abstain abstain method method method method abstain method method method abstain abstain result result abstain abstain abstain result abstain abstain result abstain abstain abstain abstain result abstain abstain abstain result abstain result abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result objective result result result method