We propose a framework for training non-autoregressive sequence-to-sequence models for editing tasks, where the original input sequence is iteratively edited to produce the output. We show that the imitation learning algorithms designed to train such models for machine translation introduces mismatches between training and inference that lead to undertraining and poor generalization in editing scenarios. We address this issue with two complementary strategies: 1) a roll-in policy that exposes the model to intermediate training sequences that it is more likely to encounter during inference, 2) a curriculum that presents easy-to-learn edit operations first, gradually increasing the difficulty of training samples as the model becomes competent. We show the efficacy of these strategies on two challenging English editing tasks: controllable text simplification and abstractive summarization. Our approach significantly improves output quality on both tasks and controls output complexity better on the simplification task. Neural sequence-to-sequence (seq2seq) models primarily developed and tested for machine translation (MT) Bahdanau et al. (2015); Vaswani et al. (2017); Gu et al. (2018) are increasingly used for other sequence transduction tasks. This paper focuses on editing tasks , such as post-editing of MT output (Simard et al., 2007), style transfer (Jin et al., 2020), or text simplification (Chandrasekar and Srinivas, 1997; Xu et al., 2015), where systems directly edit the input sequence, instead of generating the output from scratch as in MT. As illustrated in Table 1, in these tasks, there might be substantial overlap in content between inputs and outputs, and also diverse rewrites, ranging from local substitutions to more complex restructuring. Original: The Mauritshuis museum is staging an exhibition focusing on the 17th century self-portraits , highlighting the similarities and the differences between modern-day snapshots and historic works of art. Simplified: The Mauritshuis museum is now set to open an exhibit on the 17th century self-portraits . It shows the similarities and differences between modern photos and artworks. multistep, tag-then-edit approach (Alva-Manchego et al., 2017; Malmi et al., 2019; Dong et al., 2019; Mallinson et al., 2020), they can also be addressed with non-autoregressive (NAR) seq2seq models which generate their output by iteratively editing intermediate sequences (Lee et al., 2018; Gu et al., 2019; Awasthi et al., 2019; Stern et al., 2019; Chan et al., 2020). NAR models hold the promise of providing a more generic solution, where the model does not need to be tailored to a given editing task. This work is centered on the hypothesis that training NAR models for editing tasks using the same strategy as for MT leads to a mismatch between train and test settings that limits their generalization ability and output quality. Specifically, the learning algorithms designed for MT are aligned with inference strategies that generate output from an empty initial sequence. By contrast, in sequence editing tasks, the inference step is initialized instead with the original input sequence. In addition, since editing samples might range from limited lexical substitutions to more thorough rewrites, training samples cover a wide range of edit distances. During training, the loss can thus be dominated by the more distant samples leading to undertrained models and poor generalization. By contrast, the 7550 distance between input and output samples in MT is more uniform, since it always involves at least lexical translation of the input tokens. To address these issues, we introduce a new training framework, EDITINGCURRICULUM , which dynamically exposes the model to more relevant edit actions during training and exploits the full spectrum of available training samples more effectively. First, we design a new roll-in strategy, EDITING roll-in , that exposes the model to intermediate sequences that it is more likely to encounter during inference. Second, we introduce a training CURRICULUM to expose the model to training samples in order of increasing edit distance, thus gradually increasing the complexity of oracle edit operations that the model learns to imitate. We show that our approach improves the quality of outputs on two challenging English text editing tasks: controllable text simplification (TS) and abstractive summarization. It also improves the degree of TS control by generating simplified outputs that match the target reading grade level better than the baselines. We conduct an extensive analysis which supports our hypothesis, and show that the sequences generated by our training policy improve exploration during training and are easier to learn from, leading to better generalization across samples with varying edit distances. Training with curriculum further improves output quality. Model NAR edit-based models (Chan et al., 2020; Gu et al., 2019; Stern et al., 2019; Xu and Carpuat, 2021) cast sequence editing as an iterative sequence refinement problem modeled by a Markov Decision Process (cid:0) Y , A , E , R , y 0 (cid:1) . A state y = ( y 1 , y 2 , ..., y L )  Y is a sequence of tokens where each y i represents a token from the vocabulary V , L is the sequence length and y 0  Y is the initial sequence to be refined, using actions drawn from the set A . The reward R is based on the distance D between the generated output and the reference sequence y   Y : R ( y ) = D ( y, y  ) . At each decoding iteration, the model takes an input y , chooses an action a  A to refine the sequence using a policy  , resulting in state E ( y, a ) . Models differ based on the nature of edit actions used and support different operations such as insertion, deletion, reposition and substitution. We select the operations from the EDITOR model based on its competitive performance on constrained decoding tasks that require editing non-empty initial sequences (Xu and Carpuat, 2021). It is a Transformer model that uses two types of actions or edits on sequences, y : 1. The reposition operation, modeled by  rps , predicts the new position of each token in the input sequence. For each input position, the reposition policy predicts a value r that corresponds to the index of the input token to be placed at the position and 0 if the input token is to be deleted. 2. The insertion operation has two components: placeholder prediction ,  plh that predicts the number of placeholders to be inserted and token prediction ,  ins that generates the actual output tokens for each placeholder. At each decoding iteration, the model applies an action a that consists of a reposition and an insertion operation. This refinement process is repeated until two consecutive decoding iterations return the same output (Gu et al., 2019), or a preset maximum number of them is reached (Lee et al., 2018; Ghazvininejad et al., 2019). Training NAR models are typically trained via imitation learning that uses a roll-in policy and a roll-out policy. The roll-in policy is used to generate the sequences that the model learns to refine 7551 OPERATIONS Roll-In ROLL-INPOLICIES Gu et al. (2019) Insertion, Deletion Mixed y (cid:48) = {E ( y  ,  d ) ,  d   rnd } y ins = { y (cid:48) if u <  else E ( y s ,d  ) ,d     del } y del = { y s if u <  else E ( E ( y ins ,p  ) ,  t ) ,p     plh ,  t   ins } Stern et al. (2019) Insertion Expert y ins = {E ( y  ,  d ) ,  d   rnd } Ghazvininejad et al. (2019) Substitution Expert y sub = {E ( y  ,  m ) ,  m   mask } Saharia et al. (2020) Substitution (Offline) Learned y sub = {E ( y,  m ) ,  m   mask } Qian et al. (2020) Substitution Expert y sub = {E ( y,  m ) ,  m   mask } Xu and Carpuat (2021) Insertion, Reposition Learned y (cid:48) = {E ( E ( y  ,  d ) ,  p ) ,  d   rnd ,  p   per } (including deletions) y ins = { y (cid:48) if u <  else E ( y,r ) ,r   rps } y rps = { y (cid:48) if u <  else E ( E ( y,p  ) ,  t ) ,p     plh ,  t   ins } Table 2: Training Policies and Edit Operations performed by different NAR models: y s : original input sequence, y  : output sequence, y : model generated variant of reference sequence,  rnd /  masks drops/masks random words from y  according to a distribution (e.g. uniform, bernoulli, etc.),  p generates a permutation, u :  Uniform [0 , 1] ,  ins ,  plh ,  del ,  rps are insertion, placeholder prediction, deletion and reposition policies. from. A roll-out policy is then used to estimate the cost-to-go from the generated roll-in sequences to the desired output sequences. The cost-to-go is calculated by comparing the model actions to oracle demonstrations. We summarize the policies of various NAR models proposed for MT in Table 2. For EDITOR, the roll-in sequences for the reposition (or insertion) module are stochastic mixtures (parameterized by  or  ) of the output of the insertion (or reposition) module or a noised version of the output sequence. The oracle is the Levenshtein edit distance (Gu et al., 2019). The noisy sequence is generated by applying random word dropping (Gu et al., 2019) and random word shuffle (Lample et al., 2018) with a probability of 0.5 and maximum shuffle distance of 3. Figure 1 shows an example instantiation of the edit actions generated by the Levenshtein Edit Distance to transform the original input sequence (a b c d e) to the output sequence (c a t). In this example, the oracle action is to delete the tokens [b, d, e], reposition a and c and insert t at the appropriate position. The reposition and the insertion modules are trained in a supervised fashion to predict these oracle operations during training. To tailor training to editing tasks, we propose to modify the roll-in policy to better match the intermediate sequences encountered at inference, and introduce a curriculum to increase the difficulty of oracle actions learned throughout training. EDITING Roll-in Sequences generated using the roll-in policy control the search space explored during training. Those sequences should therefore be representative of the intermediate sequences generated at inference time (Ross and Bagnell, 2010). While typically, the roll-in policy is a stochastic mixture of the model and the expert demonstrations as described above, the noise incurred early on due to the large difference between the expert demonstration and the learner's policy actions may hurt overall performance (Brantley et al., 2019; He et al., 2012; Leblond et al., 2018). As we will see (5), this is what happens on editing tasks when training the model to imitate experts using learned roll-in sequences. At the same time, rolling in with expert demonstrations raises its own issues, as it can limit the exploration of the search space. Motivated by these observations, we propose a new policy, EDITING , that allows exploration by injecting noise to the input sequence to generate new intermediate sequences for training. This lets the model learn to fix errors without deviating from learning the task at hand. Figure 2 shows an example of intermediate sequences generated by our proposed roll-in policy. Different intermediate sequences encourage the model to learn differ-7552 Algorithm 1: Our proposed framework: EDITINGCURRICULUM Input: Dataset, D = { y, y  } Mi =1 , difficulty scoring function, d , and competence function, c . ent reposition and insertion edit operations starting from the same input sequence, hence enabling exploration. We modify the roll-in policies to be aligned with the editing inference process, where the reposition operation is followed by insertion on the original input sequence:  The roll-in sequence for training the reposition module,  rps , is generated by applying noise to the original source sequence y s , i.e. y rps = noise ( y s ) = {E ( E ( y s ,  d ) ,  p ) ,  d   rnd ,  p   per } . Unlike EDITOR, the random word dropping (  d   rnd ) and the word shuffling (  p   per ) are applied to the original input sequence instead of the output sequence. This aligns the training with the inference scenario where the model edits an original input sequence instead of generating an output from scratch. The roll-in sequence for training the insertion module,  ins is an intermediate sequence generated by applying the expert reposition policy to y rps , i.e. y ins = {E ( y rps , r  ) , r     rps } . The expert reposition policy corresponds to the deletion and reposition actions derived by using the levenshtein edit distance algorithm between the noisy input sequence, noise ( y s ) and the target sequence, y  . Curriculum controlled roll-out To prevent undertraining when samples with large edit distances overwhelm the loss, we use a curriculum to expose the model to easy-to-learn actions first, then gradually increase the difficulty of the edit-operations performed as the learner becomes more competent. Prior work on curriculum learning (CL) does not agree on standard measures of sample difficulty for seq2seq tasks (Kumar et al., 2019; Yao et al., 2021; Zhang et al., 2018; Zhou et al., 2020) or apply CL for the different problem of shifting the training of a Transformer model from AR to NAR regimes (Guo et al., 2020; Liu et al.). By contrast, in our settings, the Levenshtein distance provides a measure of difficulty that directly aligns with the model design and the training oracle. Resulting Algorithm Given a training dataset D = { y s , y  } Mi =1 consisting of M samples, the difficulty score d ( s i ) for each sample s i = { y si , y  i }  D is measured by the Levenshtein Distance between the input and the output sequence. The cumulative density function (CDF) of the difficulty scores results in one difficulty CDF score per sample,  d ( s i ) . At each training step t , we estimate the progress made by the learner by computing the competence of the model c ( t )  (0 , 1] as follows: c sqrt ( t ) = min   1 , (cid:115) t 1  c 20  t + c 20   where,  t defines the length of the curriculum 1 ; c 0 = 0 . 1 as in Platanios et al. (2019). Based on this competence value c ( t ) , the model is then trained on all the samples whose difficulty as measured by the Levenshtein distance between the input and the output sequence is lower than that 1 We set the curriculum length to 5 K for our experiments. We evaluate our approach on Controllable Simplification and Abstractive Summarization, two challenging sequence editing tasks that are motivated by real world information access needs. They are challenging because they require learning to perform a wide range of rewrites (from local substitution to sentence restructuring). Task Definition Given a complex text and a target grade level, the goal is to generate a simplified output that is appropriate for the desired grade level. The type of operations performed across different grade levels span sentence splitting, paraphrasing, deletion, content elaboration and substitution. Data We use English Newsela samples as extracted by Agrawal and Carpuat (2019) with 470 k/ 2 k/ 19 k for training, development and test sets respectively. Grade side-constraints are defined using a distinct special token for each grade level (from 2 to 12) and are introduced as side constraints for both the input and the output grade levels Scarton and Specia (2018). Evaluation Metrics We automatically evaluate truecased detokenized system outputs using: SARI (Xu et al., 2016), which measures the lexical simplicity based on the n-grams kept, added, and deleted by the system relative to the input and the output sequence. It computes the F1 score for the n-grams that are added ( add-F1 ). The model's deletion capability is measured by the F1 score for n-grams that are kept ( keep-F1 ) and precision for the n-grams that are deleted ( del-P ) 2 ; Pearson's correlation coefficient (PCC) between the complexity of the system and reference outputs as measured by Automatic Readability Index (ARI) (Sen-ter and Smith, 1967) and ARI-Accuracy (Heilman et al., 2008) representing the percentage of sentences where the system output grade level is within 1 grade of the reference text according to the ARI. cise summary that captures the salient ideas of the source text. It contains heavy deletions with moderate amounts of substitutions and frequent shifts caused by re-orderings. Data We use the dataset from Toutanova et al. (2016), which contains 6 K short input texts, with upto 5 summaries each. We use the same split as provided by the authors with 4937 / 448 / 786 unique input texts in the training, development and test sets respectively. The human experts were allowed to insert new words and reorder parts of the sentence when generating the summary, which makes this dataset particularly suited for abstractive summarization models. Evaluation Metrics We automatically evaluate truecased detokenized system outputs using: Rouge-L 3 (Lin, 2004). Even though it is not a summarization metric, we also report SARI to track the nature and type of edit operations performed. Given multiple references for each input text, we define the corpus level score as the arithmetic mean of automated metrics at the instance level, which is further averaged across the multiple references. Data Preprocessing We pre-process all data using Moses tools for normalization, and truecasing. We apply subword segmentation with a joint input-output byte pair encoding model with 32 , 000 operations. We use ARI to compute the input grade level at the inference time. Architecture We adopt the base Transformer architecture (Vaswani et al., 2017) with d model = 512 , d hidden = 2048 , n heads = 8 , n layers = 6 , and p dropout = 0 . 1 for all our models. We add dropout to embeddings ( 0 . 1 ) and label smoothing ( 0 . 1 ). The base EDITOR model is trained using Adam with initial learning rate of 0 . 0005 and a batch size of 16 , 000 tokens. The model is further fine-tuned on the editing task with a learning rate of 0 . 0001 . We train all our models on two GeForce GTX 1080Ti GPUs. The average training time for a single seed of AR model is  8-9 hrs and for the EDITOR model is  20-22 hrs. Fine-tuning EDITOR takes additional 5-6 hrs. Training stops after 8 checkpoints without improvement of validation perplexity. All models are implemented using the Fairseq toolkit. Models We compare our proposed approaches against the following models trained from scratch in controlled conditions: 1) AR is a auto-regressive (AR) transformer model (Scarton and Specia, 2018). 2) We train EDITOR with the dual-path roll-in policy as in Xu and Carpuat (2021), refered to as From Reference . We fine-tune EDITOR with the following policy variants: 3) From Input replaces the reference with the input for generating the initial sequence as in Agrawal et al. (2021). 4) Editing is our proposed roll-in policy. 5) Editing Curriculum , EDITCL, refers to our approach as described in 3. During inference, we start from the input sequence ( y s ), which is refined iteratively by applying a sequence of actions, as described in 2 until 1) the output sequences from two consecutive iterations are the same, or 2) the maximum number of decoding steps ( N = 10 ) is reached. The edit distance between two sequences is measured by the Levenshtein edit distance (Levenshtein et al., 1966). Controllable Simplification As can be seen in Table 3, our overall training framework, EDITCL improves over the prior training strategy for ED-ITOR From Reference  significantly for all metrics (SARI: +3.8, PCC: +0.091, ARI-Acc: +10.1%), and over the AR baseline. Ablations show that this is a combined effect of multiple factors. Dual-path roll-in, From Input improves over From Reference as expected (SARI: +1.9, PCC: +0 . 077 , ARI-Acc: +8.0%), as the roll-in sequences encountered during training are similar to those encountered during inference. Using expert roll-in ( EDITING ) performs better than using learned roll-in (dual-path roll-in) across the board, with gains of up to 3 SARI points over From Reference . Training with CL ( EDITCL ) improves over the best roll-in strategy 4 , improving the precision of deletions (+1.6) and leading to a significant improvement in SARI score (+0.7) over EDITING with no significant change in grade-specific metrics. We also report training and inference statistics. For training, we report the number of training updates to convergence, i.e. when the model achieves the best validation perplexity on the development 4 As the order of the training samples as governed by our curriculum strategy will be same for From Input, EDITING , we only report results over the best roll-in strategy. dataset. For inference, we report the average number of actions taken by the model to generate the refined output counts. Each iteration encompasses a reposition operation followed by an insertion applied to the all the tokens in the input sequence in parallel. CL reduces the average number of actions needed to generate outputs compared to EDITING , while taking only  2 K more updates during training than From Input . These results show that our roll-in policy, EDITING and the curriculum play a complementary role in improving training for editing. Abstractive Summarization On the Abstractive Summarization task (Table 4), EDITCL achieves the best performance across the board compared to alternative training strategies for EDITOR with gain of upto  4 SARI, and  3 ROUGE points. Our proposed approach improves the precision of the deletion operation ( DEL-P, +7 ). It also preserves the tokens from the source sequence that are present in the reference suggested by the improvement in KEEP-F 1( +3 . 9 ) over the EDITOR ( From Reference ) model. For completeness, we also compare our approach with systems trained in prior work: (1) ILP (Clarke and Lapata, 2008), an integer linear programing approach for deletion-based compression, (2) T3 (Cohn and Lapata, 2008), a tree transducer-based model for abstractive compression, (3) SEQ 2S EQ (Filippova et al., 2015), a neural network model for deletion-based compression, (4) NAMAS (Rush et al., 2015), a neural model for abstractive compression and summarization and (5) FELIX (Mallinson et al., 2020), a non-autoregressive approach to text editing. We use the outputs provided by Toutanova et al. (2016) for [1-4] and Mallinson et al. (2020) for [5]. We endeavored to make the comparison as fair as possible 5 , but it is not possible to have a fully controlled comparison. In particular, FELIX is trained on uncased data and generates uncased outputs, while we train and evaluate our models with truecasing. When evaluated using our pipeline, our training strategy applied to generic NAR models achieve scores that are on par with, or better than, those of dedicated summarization models (Table 5). However, this evaluation penalizes FELIX as it is trained to address the simpler problem of sum-5 We detokenized and manually checked the outputs from Mallinson et al. (2020) and corrected for de-tokenization errors such as 1. 23 to 1.23 and wanda 's to wanda's. marization on uncased text. On lower-cased outputs, our best model falls behind FELIX by 1 . 7 ROUGE points. However, FELIX has about twice as many parameters as our model and benefits from BERT pre-training (Devlin et al., 2019). As a result, this comparison confirms the promise of our approach overall. We conduct further experiments to better understand the factors that help our training strategies improve editing quality. First, we seek to measure whether our approach has the intended effect of bridging the gap between training and test for editing tasks. Figure 3 shows the distribution of oracle insertion and deletions observed when (a) training with EDITOR s default roll-in policy; (b) refining an original input sequence and (c) exposed to the model with our EDITING roll-in policy for Controllable TS. The plots show that with the default learning policy of the Editor model, the model doesn't learn to perform complex deletion operation at inference time. By contrast, our proposed roll-in exposes the model to the distribution that has higher overlap with the inference distribution as as well as additional intermediate sequences that encourages exploration during training. Training Dynamics To verify that curriculum learning helps our model better exploit its training data, we train EDITOR on x %  [0 , 100] of the data, and compare using random samples with samples ranked by increasing edit distance. Figure 4 shows the number of updates to convergence on the development dataset for controllable simplification with/without CL. Training converges early (70 iterations only) on 13% of the easiest 7556 samples with oracle edit distance between the input and the output sequence < = 2 . This supports the hypothesis that despite adding noise, our approach yields easier examples to train on. The order in which samples are presented matters, as adding batches with larger edit distance ( > 63 % data) without maintaining the order of the samples converges early. By contrast, the curriculum pacing function adds samples in order of increasing difficulty, allowing the model sufficient training time to learn from new samples while improving overall performance across metrics. We also report the learning curves when training EDITOR on the Newsela dataset in Figure 5. Training with curriculum reduces the overall loss consistently on the development dataset, leading to better generalization. Ranking Criteria We compare the edit-distance (EDITCL) with other curriculum criteria in Table 6 where the order of examples is a) random, b) controlled by the length ratio between source and target sequence (Length Ratio), c) Criteria SARI PCC % ARI-Acc Corr. governed by the difference between the source and target grade levels (Grade Difference). Our proposed criterion outperforms both task-specific ( Grade Difference ) and task-agnostic criteria ( Length Ratio ) on the Newsela Grade development set across all the metrics. Length Ratio achieves better correlation with Edit distance than Grade Difference which is also reflected by its performance (SARI: +0.3, PCC: 0.032, ARI: 0.7) on the Controllable Simplification task. This might reflect the fact that higher grade differences do not necessarily require more edits to be performed, for instance when the sentence to be simplified is already relatively simple. These mismatches do not occur when the edit distance itself is used as the sample difficulty criterion. Complementarity of roll-in and roll-out design We report the performance of the From Input model, when trained with curriculum only without the EDITING policy, i.e. EDITCLEDITING in the same Table 6. Both EDITING roll-in and curriculum controlled roll-out provides complementary advantages to the model training as removing either results in the drop in performance across all the metrics for controllable TS. However, we observe larger drop in the scores when we do not apply the EDITING policy which shows that our proposed roll-in policy is necessary to reap the benefits of curriculum learning. NAR models They have been used to enable parallel generation of output tokens for Machine translation. (Stern et al., 2019; Chan et al., 2020; Xu and Carpuat, 2021). Mallinson et al. (2020) design a custom multi-step non-autoregressive edit-based model for sequence editing where each source to-7557 % training data (x) t r a i n i ng upda t e s 0 2500 5000 7500 10000 12500 20 40 60 80 100 w/o curriculum w/ curriculum Figure 4: The sample order during training matters as training without curriculum on the same amount of data ( > = 40% ) converges early (plot on the left) and to lower performance across all metrics (plot on the right) relative to training with curriculum using the same data. ken is first tagged to represent the type of edit operation to be performed and then a secondary model is used to in-fill new tokens. The tagging and editing models are trained independently. By contrast, we propose approaches to adapt NAR models designed for MT for these tasks and train an end-to-end model to generate an edited sequence. Curriculum Learning for Sequence Refinement While curriculum learning has been applied to many tasks such as MT (Haffari, 2009; Platanios et al., 2019; Kumar et al., 2019), sentiment analysis (Sido and Konopk, 2019), natural language understanding (Xu et al., 2020), reading comprehension (Tay et al., 2019), their application to sequence refinement tasks has not been explored yet. Various strategies have been proposed to control the sample difficulty like n-gram frequency (Haffari, 2009; Platanios et al., 2019), token rarity, and sentence length (Liu et al., 2020). Chang et al. (2021) use Levenshtein edit distance as a sample difficulty criteria to order the samples for the task of data-to-text generation where the training model uses an AR seq2seq model. Instead, we focus on edit distance as a sample difficulty criteria that is directly tied to the training oracle and model design. Roll-in policies There has been a plethora of work in the Imitation learning landscape on algorithms that strike a balance between learned and expert roll-in policies (Ross et al., 2011; Venkatra-man et al., 2015; Chang et al., 2015). However, large differences in expert and learner's policy action can hurt performance (Brantley et al., 2019; He et al., 2012; Leblond et al., 2018). In our work, we propose to roll-in with noised states instead, so that the model can be exposed to mimic expert demonstrations from states that the model is more likely to encounter during inference. This paper introduced two complementary strategies to address undertraining and poor generalization when adapting NAR models to editing tasks: 1) a new roll-in policy that generates intermediate sequences that the model is likely to encounter during inference and 2) a curriculum to control the difficulty of the roll-out policy which estimates the cost-to-go from the roll-in sequences to the desired output sequences, throughout training. Together, these strategies improve output quality consistently on controllable simplification and abstractive summarization. These results open space for further research to evaluate the potential of this approach for other editing tasks (e.g., post editing, style trans-fer), and to further tailor imitation learning policies and curriculum design to these tasks. We thank Eleftheria Briakou, Khanh Nguyen, Kiant Brantley, the members of the CLIP lab at UMD, and the anonymous ARR reviewers for their helpful and constructive comments.
objective result objective result result abstain method abstain abstain abstain other abstain method abstain abstain abstain abstain abstain objective objective abstain result abstain result abstain other other other other other abstain other other other other other other other other other other objective other other other other objective other other other abstain other objective other objective objective other method other other other other objective other objective other other method other method abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method method method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other other objective other other other method other other objective objective abstain abstain other