To effectively characterize the nature of paraphrase pairs without expert human annotation, we propose two new metrics: word position deviation (WPD) and lexical deviation (LD). WPD measures the degree of structural alteration, while LD measures the difference in vocabulary used. We apply these metrics to better understand the commonly-used MRPC dataset and study how it differs from PAWS, another paraphrase identification dataset. We also perform a detailed study on MRPC and propose improvements to the dataset, showing that it improves generalizability of models trained on the dataset. Lastly, we apply our metrics to filter the output of a paraphrase generation model and show how it can be used to generate specific forms of paraphrases for data augmentation or robustness testing of NLP models. A robust understanding of semantic meaning, despite variances in sentence expression, is an integral part of natural language processing (NLP) tasks. However, many existing NLP models exhibit shortcomings in understanding real-world variations in natural language. These models are often overreliant on learned spurious correlations resulting in poor generalization (Sanchez et al., 2018; McCoy et al., 2019). This problem is challenging to address since it is difficult to distinguish spurious correlations from useful features (Gardner et al., 2021). One way of improving the performance and robustness of NLP model is to increase the size of the dataset (Hestness et al., 2017). It is possible to do so in an efficient manner through data augmentation, or the process of generating new data out of existing examples, thus creating more training data or test cases (Feng et al., 2021; Chen et al., 2021). This would also enhance the capability to detecting error in a wide range of NLP systems. We can also condition language models to generate paraphrases of input sentences (Witteveen and Andrews, 2019) through the use of large language models such as GPT (Radford et al., 2019). However, commonly used paraphrase datasets and paraphrase generation techniques that rely on such datasets suffer from several shortfalls, such as being noisy due to loose labelling in these datasets and lack of accurate, controllable generation. In this paper, we make three key contributions to address this issue. Firstly, we propose two new metrics for better understanding of paraphrase pairs: word position deviation and lexical deviation. We show, with examples, how these metrics are more effective at quantitatively capturing the linguistic characteristics of paraphrase pair than existing methods such as ROGUE-L, SELF-BLEU and edit distance. Secondly, we apply the proposed metrics to better understand the commonly used Microsoft Research Paraphrase Corpus (MRPC) (Dolan and Brockett, 2005) dataset. We also study how MRPC differs from Paraphrase Adversaries from Word Scrambling (PAWS) (Zhang et al., 2019), another paraphrase identification dataset. In the process, we perform a detailed study on MRPC and propose some revisions to the dataset. We demonstrate that this improves the quality of paraphrase identification models trained on MRPC, with higher transferability to other paraphrase identification datasets. Lastly, we demonstrate the applicability of our proposed metrics. By applying our metrics to filter the output of a paraphrase generation model, we show how it can be used to generate specific forms of paraphrases, which can be used as training data for data augmentation purposes and to generate test cases for robustness testing of NLP models. There have been several survey papers done to better understand the task of paraphrase identification and generation. A Survey of Paraphrasing and Textual Entailment Methods (Androutsopoulos and Malakasiotis, 2010) presented a comprehensive survey and review on the the aforementioned tasks. In this paper, the authors helped to properly define the tasks and identified some methods and their associated challenges. This was followed up by a more recent survey specifically on the task of paraphrase identification, A Survey on Paraphrase Recognition (Magnolini, 2014), where the focus of the survey was the performance of various statistical and non-deep learning approaches on paraphrase identification on the MRPC dataset. Additionally, in On Paraphrase Identification Corpora (Rus et al., 2014), the authors performed a survey of various paraphrase datasets, also highlighting several issues with paraphrase datasets, including MRPC, and providing some recommendations for improving the curation of paraphrase datasets. There have also been previous work on the task of better quantifying various characteristics of paraphrase pairs. In Texygen: A Benchmarking Platform for Text Generation Models (Zhu et al., 2018), SELF-BLEU was proposed to measure the diversity in text generation. However, it suffers from limitations inherent to BLEU-style metrics: it captures the differences in presence of n-grams, but not their sequence, and is thus mostly limited to capturing the differences of vocabulary, but not the overall structure of a sentence. In Paraphrasing with Large Language Models (Witteveen and Andrews, 2019), ROUGE-L is used as a measurement of paraphrase diversity, where lower ROUGE-L scores correspond to greater diversity in paraphrasing generation. However, ROUGE-L mainly measures degree of similarity in sub-sequences, but not the order in which the sub-sequences occur, and thus cannot accurately capture the possible structural differences present in paraphrase pairs. In our paper, we take a deeper look at some of the issues related to MRPC, proposing some useful improvements. We also build upon previous attempts to characterise paraphrases through the use of quantitative metrics, demonstrating how our proposed metrics can capture various different paraphrasing techniques better than previously proposed metrics. To facilitate more precise discussions in our paper, we clearly define a paraphrase as follows: Definition 1 (Paraphrase) . A sentence is a paraphrase of another sentence if they are not identical but share the same semantic meaning. Therefore, there are two distinct criteria in order to fulfill the definition of being a paraphrase pair: 1. The two sentences must have the same meaning: it is impossible to derive different information from a paraphrase of a sentence. Where two sentences are not certain to have exactly the same meaning, a common interpretation of both sentences should be the same in order for it to be a reasonable paraphrase. This also implies that both sentences in a paraphrase pair necessarily entail each other. 2. The two sentences must not be identical, for example having lexical differences (differ-ences in vocabulary) or structural differences (differences in word order, punctuation and syntax). In A Survey of Paraphrasing and Textual Entailment Methods (Androutsopoulos and Malakasiotis, 2010), the following example is provided, which we shall discuss: It is argued that sentence 3 is not a precise paraphrase of sentences 1 and 2 as it is not stated precisely in sentence 3 that the bridge was completed. For the purposes of our discussion, we would consider sentence 3 a reasonable paraphrase as well as it is very likely that all three sentences would be interpreted in the same way, and thus share the same semantic meaning based on the most common interpretation of the sentences. These examples illustrate that it is non-trivial to precisely define what is a paraphrase pair, as there is some variance (depending on subjective interpretation) on what would be a precise paraphrase. This problem is observed to have caused issues due to the imprecise definitions used while creating paraphrase datasets, such as the MRPC dataset which is very widely used. By adhering strictly to the definition of a paraphrase as detailed in this section, we hope to better facilitate discussion throughout the paper. In this paper, we will utilize and compare two commonly used paraphrase datasets, MRPC and PAWS. The Microsoft Research Paraphrase Corpus (MRPC) is a corpus consists of sentence pairs collected from web news articles (Dolan and Brockett, 2005). This dataset is widely used as a benchmark for the paraphrase identification task. It can be used directly or indirectly as part of the GLUE benchmark (Wang et al., 2019). In particular, as part of the GLUE benchmark, the dataset has been used for training and evaluation in more than 50 research papers as can be determined from the GLUE leaderboard 1 . It is also less commonly used as a paraphrase generation dataset, in works such as (Huang and Chang, 2021). MRPC contains 4076 training and 1725 test examples. The Paraphrase Adversaries from Word Scrambling (PAWS) is a dataset contains sentence pairs extracted from Wikipedia and the Quora Question Pairs (QQP) dataset (Zhang et al., 2019). While it is less commonly used than MRPC, it is a high quality and larger dataset, and is used in a number of papers such as (Yu and Ettinger, 2021), (Tu et al., 2020) and (Chen et al., 2020) for the purpose of paraphrase identification. PAWS contains 49,401 training, 8000 development and 8000 test examples. Our objective is to comprehensively evaluate the diverse linguistic phenomena involved in paraphrasing, which can include techniques such as synonym substitution, negation, diathesis alternation, coordination changes and more. We can broadly classify these techniques into the use of structural alternations and lexical alternations to achieve paraphrasing. Thus, to better characterise a paraphrase pair, we propose two metrics: word position deviation and lexical deviation . These two metrics are introduced so as to provide a quantitative understanding on what type of paraphrase it is along the two types 1 https://gluebenchmark.com/leaderboard of changes. A key design consideration of these metrics is the need to be able to capture the extents of structural and lexical alterations in an efficient manner, without resorting to costly human annotation or large amounts of computation. We will use these metrics to provide a good understanding of the characteristics of paraphrase pairs both at a individual (paraphrase pair) level and at an aggregate level over the whole dataset. In addition, we apply these metrics to filter outputs from paraphrase generation systems to select for specific types of paraphrases. In this section, we define some terms that will be used across various metrics computations. Let s 1 and s 2 denote two sentences. We will also refer to the pair of sentences ( s 1 , s 2 ) as a paraphrase pair. Definition 4.1 (Set of common words) . The set of common words C ( s 1 ,s 2) of a paraphrase pair is the set of words, in uncased lemmatized form, which occurs in both s 1 and s 2 . Definition 4.2 (Set of all words) . The set of all words A ( s 1 ,s 2) of a paraphrase pair is the complete set of words, in uncased lemmatized form, which occurs in either or both sentences s 1 and s 2 . Thus, given two sentences:  s 1 : Yesterday, Bob met Tom at the store. s 2 : Tom met Bob yesterday while they were at the store. C ( s 1 ,s 2) : { yesterday, bob, meet, tom, at, the, store }  A ( s 1 ,s 2) : { yesterday, bob, meet, tom, at, the, store, while, they, be } We will also use the notation NC ( s 1 ,s 2) to refer to the size of set C ( s 1 ,s 2) and NA ( s 1 ,s 2) to refer to the size of set A ( s 1 ,s 2) . We use NC and NA for short when it is obvious which statements s 1 and s 2 we are referring to. For a word W and a sentence s , we denote by N s ( W ) the number of times that the word W appears in the sentence s . We propose the word position deviation (WPD) of a paraphrase pair as a metric that effectively captures the degree of deviation in the structure of paraphrased sentences by looking at changes in word positions. WPD can be intuitively understood as the mean of how much words shift in position after a paraphrase. We find that this proposed metric is effective in identifying the amount of structural alterations present in paraphrase pairs. To properly define WPD, we first introduce the concept of normalized word position in a paraphrase pair. Definition 4.3 (Normalized Word Position) . Let s be a sentence and W be a word. For 1  n  N s ( W ) , the normalized word position  s,n ( W ) of n -th appearance of W in s is its index divided by the index of the last word. Thus, a normalized word position value ranges from the first word in the sentence having a value of 0.0 and last word having value of 1.0. For example, if the second appearance of W has index a and the last word has index b in the sentence s , then  s, 2 ( W ) = a/b . In WPD, we consider the mean differences between the normalized word positions. For any given word that is common in both sentences in a paraphrase pair ( s 1 , s 2 ) , we can calculate the relative position shift as the difference in normalized word position. Definition 4.4 (Relative Position Shift) . The relative position shift of a word W with respect to sentence s 1 in paraphrase pair ( s 1 , s 2 ) is denoted as  s 1 ,s 2 ( W ) , only defined for words in C ( s 1 ,s 2) , and has the expression  s 1 ,s 2 ( W ) = N s 1 ( W ) (cid:88) n =1 min 1  k  N s 2 ( W ) |  s 1 ,n ( W )   s 2 ,k ( W ) | N s 1 ( W ) . (1) For each occurrence of W in s 1 , we calculate the smallest difference between its normalized word position and that of the occurrences of W in s 2 . We then average these smallest differences over all occurrence of W in s 1 to get the relative position shift of W with respect to s 1 in paraphrase pair ( s 1 , s 2 ) . In a simple case with only one occurrence of W in each sentence, this reduces to the distance between  s 1 , 1 ( W ) and  s 2 , 1 ( W ) , which is  s 1 ,s 2 ( W ) = |  s 1 , 1 ( W )   s 2 , 1 ( W ) | . (2) To the concepts described above, a simple example is provided in Figure 1 below. close to 1.0. Conversely, if the word W is near the start of s 1 and near the start of the s 2 ,  s 1 ,s 2 ( W ) is close to 1.0. In a generalised case where there can be multiple occurrences of W can be present in s 1 or s 2 , the mean distance between one occurrence and the nearest occurrence in the other sentence is considered. However, such instances are much rarer. We illustrate the handling of using a real example, showing how the word his occurs twice, resulting in a mean  ( his ) of 0.263. Definition 4.5 (Word Position Deviation) . Let ( s 1 , s 2 ) be a paraphrase pair. The WPD of ( s 1 , s 2 ) , denoted as  pos ( s 1 , s 2) , is the mean of all the relative position shifts of all the words in the set C ( s 1 ,s 2) , namely,  pos ( s 1 , s 2) = 1 NC (cid:88) W C max {  s 1 ,s 2 ( W ) ,  s 2 ,s 1 ( W ) } . (3) Below are additional examples of the WPD computation on paraphrases in the MRPC dataset. To aid visualization of what the metric measures, the common words are underlined and coloured to aid comparison. We propose lexical deviation (LD), a metric that effectively captures the degree of deviation in the vocabulary used between the sentences in a paraphrase pair. We find that the proposed metric is effective in identifying and ranking paraphrase pairs 8595 from various datasets according to meaningful differences in their usage of lexical changes to perform paraphrasing. Definition 4.6 (Lexical Deviation) . Let ( s 1 , s 2 ) be a paraphrase pair. The lexical deviation  lex ( s 1 , s 2 ) for a paraphrase pair ( s 1 , s 2 ) is defined by  lex ( s 1 , s 2 ) = 1  NCNA . For a case where there is complete reuse of words (in other words, NC = NA ), the metric will compute to 0.0. Likewise, in a case where there is no reuse of words, the metric computes to 1.0. For the purpose of computing the total set of words and the set of common words, we consider words that are the same after lemmatization (ig-noring capitalization) to be the same word. Therefore, we do not consider words that are of different forms (e.g. tense) and capitalization to be different words. This allows our metric to more accurately capture the range of vocabulary used. As word forms tend to vary when used as part of different sentence structures, we do not wish to capture that in this metric, which focuses on the diversity of vocabulary (using of different words), and not the grammatical usage of a word. In addition, we consider changes in capitalization a trivial paraphrase, and hence do not consider it in this metric. To demonstrate the applicability of our proposed metrics of WPD and LD, we compare them against other metrics with similar purposes: ROGUE-L (Lin, 2004), SELF-BLEU (Zhu et al., 2018) and DamerauLevenshtein edit distance (Levenshtein, 1965). In the examples below, we show that with WPD and LD, we can effectively distinguish between different types of paraphrases that have similar scores via various other metrics. two paraphrases can have very similar ROGUE-L scores of 0.76 and 0.75, where ROUGE-L primarily measures the degree of sub-string similarity (longest common sub-strings). However, with WPD, we are able to additionally distinguish the degree in which the similar sub-strings have been shuffled in position, which is a structural alteration to the sentence. In Example Pair 2 (Figure 4), we again show that two paraphrases can have very similar SELF-BLEU scores of 0.60 and 0.59, where SELF-BLEU primarily measures the degree n-gram overlap. However, similar to Example Pair 1, in one of the paraphrases, the two "halves" of the sentence has been swapped in position, and this structural alteration is captured by the WPD score. Lastly, in Example Pair 3 (Figure 5), we show that two paraphrases can have very similar DamerauLevenshtein edit distance, but feature two completely different types of paraphrasing method. Using WPD, we are able to obtain an aggregate view of both the MRPC and PAWS datasets. We see that both datasets feature similar distributions of structural paraphrasing, where the average amount of structural paraphrasing is fairly low and MRPC features more structural paraphrasing compared to 8596 PAWS. A visualization is provided in Figure 6 below. Hence, we would expect the MRPC dataset to be somewhat more diverse in structural paraphrases as compared to PAWS. Using LD, we are able to obtain an aggregate view of both the MRPC and PAWS datasets to see that both datasets feature a very different distribution of lexical paraphrasing. A visualization is provided in Figure 7 below. MRPC features a large amount of lexical paraphrasing, in contrast to PAWS where lexical paraphrasing is almost absent. Hence, we would expect the MRPC dataset to be substantially more diverse in having different examples of lexical paraphrases as compared to PAWS. We investigated the source of high LD in MRPC and determined that the reason is due to large inconsistencies in entities, such as named entities and quantities, present in MRPC paraphrase pairs. We can see that many of the examples at the high-end of lexical deviation are not reasonable paraphrases of each other as they contain extremely different information in each sentence. When used as training data for paraphrase identification or generation tasks, this can introduce undesired behaviour into models. For example, this can make paraphrase generation models more prone to "hallucinating" additional information in paraphrases, while paraphrase identification models are less able to detect such inconsistencies. Hence, this motivates us to more closely inspect the quality and consistency of labels in the MRPC dataset, and then propose improvements. Despite its wide usage as a benchmark for paraphrase identification, the labels in the MRPC dataset are not of a consistently high quality. This is a result of the annotation process used to create the MRPC dataset. The annotation process used for MRPC, as described in the paper Automatically constructing a corpus of sentential paraphrases (Dolan and Brock-ett, 2005), is as follows: a collection of news articles is collected from the web over a 2-year period, and candidates for paraphrase pairs are extracted using automated approaches, followed by human evaluation used to determine if two similar sentences are paraphrases. However, the instructions given to the human annotators of the pairs were "ill-defined". Compounding the issue is that several classes of named entities in the text were replaced by generic tags, introducing large amounts of ambiguity. As a result, the annotators labelled sentences with very inconsistent entities as valid paraphrases, leading to a relatively large number of sentences inside that are not in fact reasonable paraphrases, despite being labelled as such. Thus, models that perform well on MRPC may not able to correctly identify paraphrases in a precise manner. We can show this in Section 5.2.2, where a state-of-the-art language model that performs well on MRPC has nearly random performance on PAWS, despite both being paraphrase identification datasets. To illustrate this issue, we use an example of a sentence pair, labelled as a paraphrase, from the MRPC dataset: 1. The stock rose $2.11, or about 11 percent, to close Friday at $21.51 on the New York Stock Exchange. 2. PG&E Corp. shares jumped $1.63 or 8 percent to $21.03 on the New York Stock Exchange on Friday. In this example, which is labelled as a paraphrase-pair, there are a total of 9 entities across the paraphrase pair, but only 2 ("the New York 8597 Stock Exchange" and "Friday") are present across the two. In other words, there is a great inconsistency in the entities present between each of the paraphrase pairs. In this case, this results in a large discrepancy in the information contained in each sentence, and thus the two sentences are not in fact paraphrases despite being labelled as such in MRPC. In MRPC, there are a total of 3900 paraphrase pairs. Of those, 3016 (77%) have at least 1 inconsistent entity. Thus, this is a common issue in MRPC. With the aim to improve the precision of sentence pairs labelled as paraphrases in MRPC, we proposed some amendments to MRPC, including the following specific objectives: 1. Automatically correcting the inconsistency in entities; 2. Rectifying the labels where automated correction is not possible. Our process to achieve this has two main steps. First, we search for inconsistent examples where the inconsistency is limited to singular instances of any type of quantity. For example, one instance of a monetary value that differs between two sentences in a paraphrase pair. Next, when a match is found, we proceed with a to correct the paraphrase. In this specific scenario, as we know that both values share the same type, we can correct one of the values to be identical to the instance in the other sentence, making it a more precise paraphrase. In order to avoid being overly zealous in this replacement, we inspect the most frequent replacements to ensure that no unintended replacements occur. Of the 3016 inconsistent paraphrase pairs in MRPC, 476 (16%) can be corrected using our approach. For the rest of the paraphrase pairs that we cannot correct, we label them as non-paraphrases. After the corrections, 2064 (53%) out of the original 3900 paraphrase pairs are relabelled as non-paraphrases. This also changes the ratio of paraphrase:non-paraphrase in MRPC from approximately 8:5 to approximately 4:8. We term this revised version of MRPC as MRPC-R1. To illustrate the corrections to text performed during the creation of MRPC-R1, a few examples are shown in the table below: Figure 9: Correcting some examples from MRPC 5.2.2 Evaluating Changes to MRPC In order to evaluate the differences in quality of the datasets, we compare the transferability of a model trained on MRPC and MRPC-R1 to the PAWS test set. Our training setup is as follows: We used a state-of-the-art DeBERTa (He et al., 2021) pretrained langauage model and fine-tuned it on each of the following: MRPC training set, MRPC-R1 training set, and lastly for a baseline, the PAWS training set). We performed the training using the HuggingFace Transformers library (Wolf et al., 2020) and PyTorch (Paszke et al., 2019), learning rate of 1e-5, and the Adam optimizer (Kingma and Ba, 2015). For MRPC and MRPC-R1, we use a batch size of 32, and for PAWS, which has a much larger training set, we use a batch size of 128. We did not perform extensive hyper-parameter tuning. We tested two variations of the DeBERTa model: DeBERTa-base (140M parameters) and DeBERTa-large (400M parameters) Each of the models are evaluated every 50 steps on the PAWS development set, and the best model checkpoint is evaluated against the PAWS test set. We report the results below (median from 5 runs). From our results, we can see that training on MRPC-R1 results in much better scores on the PAWS test set for both models. Additionally, if we use the more powerful DeBERTa-large model, the model overfits more on MRPC training data. Thus, DeBERTa-large scores lower than DeBERTa-base on the PAWS test set. However, DeBERTa-large performs better than DeBERTa-base when trained on MRPC-R1, showing that more powerful 8598 models benefit more from MRPC-R1. Thus, we can see that that MRPC-R1 has greater transferability to the PAWS test set. These results demonstrate that we have increased the generalization ability of the trained model through the improving the consistency and quality of the labels in MRPC. To demonstrate the applicability of our metrics to filter and thus control the output from a paraphrase generation model, we combine the paraphrase pairs from MRPC-R1 and PAWS to form a corpus to train a sequence-to-sequence T5 (Raf-fel et al., 2020) transformer language model to generate paraphrases. We performed the training using the HuggingFace Transformers library and PyTorch, using the the pretrained T5-large model (770M parameters). We performed training for a total of 10 epochs with a batch size of 16, learning rate of 1e-5, the Adam optimizer and did not perform extensive hyper-parameter tuning. By using WPD and LD, we are able to effectively filter for specific types of paraphrases. In the following example, we pass " I keep a glass of water next to my bed when I sleep. " as an input to be paraphrased by the model. Some of the outputs are sampled and ranked below according to WPD, showing how WPD can be used to select paraphrases with varying extents of structural paraphrases, and the results can be seen in the table below: Generated Paraphrase WPDI keep a glass of water beside my bed when I sleep. We can also do the same for LD, where we can see that the lower the the extent of word overlap between the original and paraphrase, the greater the LD value. Words are marked with italics to visually indicate words that have changed from the source sentence. The results can be seen in the table below: Generated Paraphrase LD When I sleep I keep a glass of water next to my bed. Thus, we can use WPD, LD, or a combination of both to select specific types of paraphrases, therefore efficiently obtaining specific variations of data for data augmentation or robustness testing purposes. To the best of our knowledge, we do not introduce any ethical concerns in this work. Our work is based on the existing MRPC and PAWS datasets, which are sampled from online news articles as well as Wikipedia. Hence we expect our findings to generalize well to other English datasets in the general domain. Generalization of our work to domains where usage of language is markedly different (for example, in some forms of technical writing) is not certain. When our proposed metrics are used in conjunction with other technology (such as large generative language models), it does not affect the existing ethical considerations of using those technology. In our paper, we have proposed two new metrics to better understand paraphrase pairs: word position deviation (WPD) and lexical deviation (LD). We have applied these metrics to better understand the MRPC and PAWS datasets, and also to filter the output of a paraphrase generation model to obtain specific forms of paraphrases. However, our metrics still have some limitations, which can be address in future work. Although we are able to measure the extent of structural and lexical alterations, we cannot determine the fine-grained type of alterations that is being made, for example, a specific form of structural alteration or word substitution. We anticipate that improvements in this area would be valuable to improve our ability to effectively characterize various properties of paraphrases, leading to better data augmentation and robustness testing approaches that eventually resulting in better performing NLP systems.
objective abstain objective objective result abstain abstain abstain abstain abstain abstain abstain method abstain objective objective result abstain abstain objective objective objective objective other other other method other other other other other other other objective objective method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method method method method abstain abstain method abstain abstain abstain abstain abstain method method method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain objective result abstain method result