Neural network-based sequence-to-sequence (seq2seq) models strongly suffer from the low-diversity problem when it comes to open-domain dialogue generation. As bland and generic utterances usually dominate the frequency distribution in our daily chitchat, avoiding them to generate more interesting responses requires complex data filtering, sampling techniques or modifying the training objective. In this paper, we propose a new perspective to diversify dialogue generation by leveraging non-conversational text. Compared with bilateral conversations, nonconversational text are easier to obtain, more diverse and cover a much broader range of topics. We collect a large-scale nonconversational corpus from multi sources including forum comments, idioms and book snippets. We further present a training paradigm to effectively incorporate these text via iterative back translation. The resulting model is tested on two conversational datasets and is shown to produce significantly more diverse responses without sacrificing the relevance with context. Seq2seq models have achieved impressive success in a wide range of text generation tasks. In open-domain chitchat, however, people have found the model tends to strongly favor short, generic responses like I don't know or OK (Vinyals and Le, 2015; Shen et al., 2017a). The reason lies in the extreme one-to-many mapping relation between every context and its potential responses (Zhao et al., 2017; Su et al., 2018). Generic utterances, which can be in theory paired with most context, usually dominate the frequency distribution in the dialogue training corpus and thereby pushes the model to  Equal contribution. blindly produce these safe, dull responses (Su et al., 2019b; Csaky et al., 2019) Current solutions can be roughly categorized into two classes: (1) Modify the seq2seq itself to bias toward diverse responses (Li et al., 2016a; Shen et al., 2019a). However, the model is still trained on the limited dialogue corpus which restricts its power at covering broad topics in open-domain chitchat. (2) Augment the training corpus with extra information like structured world knowledge, personality or emotions (Li et al., 2016b; Dinan et al., 2019), which requires costly human annotation . In this work, we argue that training only based on conversational corpus can greatly constrain the usability of an open-domain chatbot system since many topics are not easily available in the dialogue format. With this in mind, we explore a cheap way to diversify dialogue generation by utilizing large amounts of non-conversational text . Compared with bilateral conversations, non-conversational text covers a much broader range of topics, and can be easily obtained without further human annotation from multiple sources like forum comments, idioms and book snippets. More importantly, nonconversational text are usually more interesting and contentful as they are written to convey some specific personal opinions or introduce a new topic, unlike in daily conversations where people often passively reply to the last utterance. As can be seen in Table 1, the response from the daily conversation is a simple comfort of Head pat. Nonconversational text, on the contrary, exhibit diverse styles ranging from casual wording to poetic statements, which we believe can be potentially utilized to enrich the response generation. To do so, we collect a large-scale corpus containing over 1M non-conversational utterances from multiple sources. To effectively integrate these utterances, we borrow the back translation idea from unsupervised neural machine translation (Sennrich et al., 2016; Lample et al., 2018b) and treat the collected utterances as unpaired responses. We first pre-train the forward and backward transduction model on the parallel conversational corpus. The forward and backward model are then iteratively tuned to find the optimal mapping relation between conversational context and non-conversational utterances (Cotterell and Kreutzer, 2018). By this means, the content of non-conversational utterances is gradually distilled into the dialogue generation model (Kim and Rush, 2016), enlarging the space of generated responses to cover not only the original dialogue corpus, but also the wide topics reflected in the non-conversational utterances. We test our model on two popular Chinese conversational datasets weibo (Shang et al., 2015a) and douban (Wu et al., 2017). We compare our model against retrieval-based systems, style-transfer methods and several seq2seq variants which also target the diversity of dialogue generation. Automatic and human evaluation show that our model significantly improves the responses' diversity both semantically and syntactically without sacrificing the relevance with context, and is considered as most favorable judged by human evaluators 1 . The tendency to produce generic responses has been a long-standing problem in seq2seq-based open-domain dialogue generation (Vinyals and Le, 2015; Li et al., 2016a). Previous approaches to alleviate this issue can be grouped into two classes. The first class resorts to modifying the seq2seq architecture itself. ing objective to mutual information maximization and rely on continuous approximations or policy gradient to circumvent the non-differentiable issue for text. Li et al. (2016d); Serban et al. (2017a) treat open-domain chitchat as a reinforcement learning problem and manually define some rewards to encourage long-term conversations. There is also research that utilizes latent variable sampling (Ser-ban et al., 2017b; Shen et al., 2018b, 2019b), adversarial learning (Li et al., 2017; Su et al., 2018), replaces the beam search decoding with a more diverse sampling strategy (Li et al., 2016c; Holtzman et al., 2019) or applies reranking to filter generic responses (Li et al., 2016a; Wang et al., 2017). All of the above are still trained on the original dialogue corpus and thereby cannot generate out-of-scope topics. The second class seeks to bring in extra information into existing corpus like structured knowledge (Zhao et al., 2018; Ghazvininejad et al., 2018; Dinan et al., 2019), personal information (Li et al., 2016b; Zhang et al., 2018a) or emotions (Shen et al., 2017b; Zhou et al., 2018). However, corpus with such annotations can be extremely costly to obtain and is usually limited to a specific domain with small data size. Some recent research started to do dialogue style transfer based on personal speeches or TV scripts (Niu and Bansal, 2018; Gao et al., 2019; Su et al., 2019a). Our motivation differs from them in that we aim at enriching general dialogue generation with abundant non-conversational text instead of being constrained on one specific type of style. Back translation is widely used in unsupervised machine translation (Sennrich et al., 2016; Lample et al., 2018a; Artetxe et al., 2018) and has been recently extended to similar areas like style transfer (Subramanian et al., 2019), summarization (Zhao et al., 2019) and data-to-text (Chang et al., 2020). To the best of our knowledge, it has never been applied to dialogue generation yet. Our work treats the context and non-conversational text as unpaired source-target data. The back-translation idea is naturally adopted to learn the mapping between them. The contents of nonconversational text can then be effectively utilized to enrich the dialogue generation. We would like to collect non-conversational utterances that stay close with daily-life topics and can be potentially used to augment the response space. The utterance should be neither too long nor too short, similar with our daily chitchats. Therefore, we collect data from the following three sources: 1. Forum comments. We collect comments from zhihu 2 , a popular Chinese forums. Selected comments are restricted to have more than 10 likes and less than 30 words 3 . 2. Idioms. We crawl idioms, famous quotes, proverbs and locutions from several websites. These phrases are normally highly-refined and graceful, which we believe might provide a useful augmentation for responses. 3. Book Snippets. We select top 1,000 favorite novels or prose from wechat read 4 . Snippets highlighted by readers, which are usually quintessential passages, and with the word length range 10-30 are kept. We further filter out sentences with offensive or discriminative languages by phrase matching against a large blocklist. The resulting corpus contains over 1M utterances. The statistics from each source are listed in Table 2. 4 Approach Let D = { ( X 1 , Y 1 ) , ( X 2 , Y 2 ) , . . . , ( XN , YN ) } denote the parallel conversational corpus. X i is the context and Y i is the corresponding response. DT = { T 1 , T 2 , . . . , TM } denotes our collected corpus where T i is a non-conversational utterance. As the standard seq2seq model trained only on D tends to generate over-generic responses, our purpose is to diversify the generated responses by leveraging the non-conversational corpus DT , which are semantically and syntactically much richer than 2 https://www.zhihu.com 3 The posts are usually very long, describing a specific social phenomenon or news event, so building parallel conversational corpus from post-comment pairs is difficult. Nonetheless, these high-liked comments are normally high-quality themselves and can be used to augment the response space. 4 https://weread.qq.com/ responses contained in D . In the following section, we first go through several baseline systems, then introduce our proposed method based on back translation. The first approach we consider is a retrieval-based system that considers all sentences contained in DT as candidate responses. As the proportion of generic utterances in DT is much lower than that in D , the diversity will be largely improved. Standard retrieval algorithms based on context-matching (Wu et al., 2017; Bartl and Spanakis, 2017) fail to apply here since non-conversational text does not come with its corresponding context. Therefore, we train a backward seq2seq model on the parallel conversational corpus D to maximize p ( X i | Y i ) . The score assigned by the backward model, which can be seen as an estimation of the point-wise mutual information, is used to rank the responses (Li et al., 2016a) 5 . The major limitation of the retrieval-based system is that it can only produce responses from a finite set of candidates. The model can work well only if an appropriate response already exists in the candidate bank. Nonetheless, due to the large size of the non-conversational corpus, this approach is a very strong baseline. The second approach is to take a weighted average score of a seq2seq model trained on D and a language model trained on DT when decoding responses. The idea has been widely utilized on domain adaptation for text generation tasks (Koehn and Schroeder, 2007; Wang et al., 2017; Niu and Bansal, 2018). In our scenario, basically we hope the generated responses could share the diverse topics and styles of the non-conversational text, yet stay relevant with the dialogue context. The seq2seq model S 2 S is trained on D as an indicator of how relevant each response is with the context. A language model L is trained on DT to measure how the response matches the domain of DT . The decoding probability for generating word w at time step t is assigned by: p t ( w ) = S 2 S t ( w ) + (1   ) L t ( w ) (1) 5 The backward seq2seq model measures the context relevance better than forward models since the latter highly biases generic utterances (Li et al., 2016a; Zhang et al., 2018b) Figure 1: Comparison of four approaches leveraging the non-conversational text. where  is a hyperparameter to adjust the balance between the two. Setting  = 1 will make it degenerate into the standard seq2seq model while  = 0 will totally ignore the dialoge context. The third approach is based on multi-task learning. A seq2seq model is trained on the parallel conversational corpus D while an autoencoder model is trained on the non-parallel monologue data DT . Both models share the decoder parameters to facilitate each other. The idea was first experimented on machine translation in order to leverage large amounts of target-side monolingual text (Luong et al., 2016; Sennrich et al., 2016). Luan et al. (2017) extended it to conversational models for speaker-role adaptation. The intuition is that by tying the decoder parameters, the seq2seq and autoencoder model can learn a shared latent space between the dialogue corpus and non-conversational text. When decoding, the model can generate responses with features from both sides. Finally, we consider the back translation technique commonly used for unsupervised machine translation (Artetxe et al., 2018; Lample et al., 2018a). The basic idea is to first initialize the model properly to provide a good starting point, then iteratively perform backward and forward translation to learn the correspondence between context and unpaired non-conversational utterances. Initialization Unlike unsupervised machine translation, the source and target side in our case come from the same language, and we already have a parallel conversational corpus D , so we can get rid of the careful embedding alignment and autoencoding steps as in Lample et al. (2018b). For the initialization, we simply train a forward and backward seq2seq model on D . The loss function is: EX i ,Y i D  log P f ( Y i | X i )  log P b ( X i | Y i ) (2) where P f and P b are the decoding likelihood defined by the forward and backward seq2seq model respectively. We optimize Eq. 2 until convergence. Afterwards, the forward and backward seq2seq can learn the backbone mapping relation between a context and its response in a conversational structure. Backward After the initialization, we use the backward seq2seq to create pseudo parallel training examples from the non-conversational text DT . The forward seq2seq is then trained on the pseudo pairs. The objective is to minimize: ET i D T  log P f ( T i | b ( T i )) b ( T i ) = arg max u P b ( u | T i ) (3) where we approximate the arg max function by using a beam search decoder to decode from the backward model P b ( u | T i ) . Because of the non-differentiability of the arg max operator, the gradient is only passed through P f but not P b 6 . As P b is already well initialized by training on the parallel corpus D , the back-translated pseudo 6 As also noted in Lample et al. (2018b), backpropagating further through P b brings no improvement. ( Inilialization ) Train by minimizing Eq. 2 until convergence; for i=1 to N do ( Backward ) Train by minimizing Eq. 3 until convergence; ( Forward ) Train by minimizing Eq. 4 until convergence; end Algorithm 1: Model Training Process pair { b ( T i ) , T i } can roughly follow the typical human conversational patterns. Training P f on top of them will encourage the forward decoder to generate utterances in the domain of T i while maintaining coherent as a conversation. Forward The forward translation follows a similar step as back translation. The forward seq2seq P f translates context into a response, which in return form a pseudo pair to train the backward model P b . The objective is to minimize: EX i D  log P b ( X i | f ( X i )) f ( X i ) = arg max v P f ( v | X i ) (4) where the arg max function is again approximated with a beam search decoder and the gradient is only backpropagated through P b . Though X i has its corresponding Y i in D , we drop Y i and instead train on forward translated pseudo pairs { X i , f ( X i ) } . As P f is trained by leveraging data from DT , f ( X i ) can have superior diversity compared with Y i . The encoder parameters are shared between the forward and backward models while decoders are separate. The backward and forward translation are iteratively performed to close the gap between P f and P b (Hoang et al., 2018; Cotterell and Kreutzer, 2018). The effects of non-conversational text are strengthened after each iteration. Eventually, the forward model will be able to produce diverse responses covering the wide topics in DT . Algorithm 1 depicts the training process. We conduct our experiments on two Chinese dialogue corpus Weibo (Shang et al., 2015b) and Douban (Wu et al., 2017). Weibo 7 is a popular Twitter-like microblogging service in China, on which a user can post short messages, and other 7 http://www.weibo.com/ users make comment on a published post. The post-comment pairs are crawled as short-text conversations. Each utterance has 15.4 words on average and the data is split into train/valid/test subsets with 4M/40k/10k utterance pairs. Douban 8 is a Chinese social network service where people can chat about different topics online. The original data contains 1.1M multi-turn conversations. We split them into two-turn context-response pairs, resulting in 10M train, 500k valid and 100K test samples. For all models, we use a two-layer LSTM (Hochre-iter and Schmidhuber, 1997) encoder/decoder structure with hidden size 500 and word embedding size 300. Models are trained with Adam optimizer (Kingma and Ba, 2015) with an initial learning rate of 0.15. We set the batch size as 256 and use the gradients clipping of 5. We build out vocabulary with character-based segmentation for Chinese. For non-Chinese tokens, we simply split by space and keep all unique tokens that appear at least 5 times. Utterances are cut down to at most 50 tokens and fed to every batch. We implement our models based on the OpenNMT toolkit (Klein et al., 2017) and other hyperparameters are set as the default values. We compare our model with the standard seq2seq and four popular variants which were proposed to improve the diversity of generated utterances. All of them are trained only on the parallel conversational corpus: Standard The standard seq2seq with beam search decoding (size 5). MMI The maximum mutual information decoding which reranks the decoded responses with a backward seq2seq model (Li et al., 2016a). The hyperparameter  is set to 0.5 as suggested. 200 candidates per context are sampled for re-ranking Diverse Sampling The diverse beam search strategy proposed in Vijayakumar et al. (2018) which explicitly controls for the exploration and exploitation of the search space. We set the number of groups as 5,  = 0 . 3 and use the Hamming diversity as the penalty function as in the paper. 8 https://www.douban.com/group Nucleus Sampling Proposed in Holtzman et al. (2019), it allows for diverse sequence generations. Instead of decoding with a fixed beam size, it samples text from the dynamic nucleus. We use the default configuration and set p = 0 . 9 . CVAE The conditional variational autoencoder (Serban et al., 2017b; Zhao et al., 2017) which injects diversity by imposing stochastical latent variables. We use a latent variable with dimension 100 and utilize the KL-annealing strategy with step 350k and a word drop-out rate of 0.3 to alleviate the posterior collapse problem (Bowman et al., 2016). Furthermore, we compare the 4 approaches mentioned in  4 which incorporate the collected nonconversational text: Retrieval-based (  4.1) Due to the large size of the non-conversational corpus, exact ranking is extremely slow. Therefore, we first retrieve top 200 matched text with elastic search based on the similarity of Bert embeddings (Devlin et al., 2019). Specifically, we pass sentences through Bert and derive a fixed-sized vector by averaging the outputs from the second-to-last layer (May et al., 2019) 9 . The 200 candidates are then ranked with the backward score 10 . Weighted Average (  4.2) We set  = 0 . 5 in eq. 1, which considers context relevance and diversity with equal weights. Multi-task ((  4.3)) We concatenate each context-response pair with a non-conversational utterance and train with a mixed objective of seq2seq and autoencoding (by sharing the decoder). Back Translation (  4.4) We perform the iterative backward and forward translation 4 times for both datasets. We observe the forward cross entropy loss converges after 4 iterations. As for the experiment results, we report the automatic and human evaluation in  6.1 and  6.2 respectively. Detailed analysis are shown in  6.3 to elaborate the differences among model performances and some case studies. 10 This makes it similar to MMI reranking, whose 200 candidates are from seq2seq decodings instead of top-matched non-conversational utterances. Evaluating dialogue generation is extremely diffi-cult. Metrics which measure the word-level overlap like BLEU (Papineni et al., 2002) have been widely used for dialogue evaluation. However, these metrics do not fit into our setting well as we would like to diversify the response generation with an external corpus, the generations will inevitably differ greatly from the ground-truth references in the original conversational corpus. Though we report the BLEU score anyway and list all the results in Table 3, it is worth mentioning that the BLEU score itself is by no means a reliable metric to measure the quality of dialogue generations. Diversity Diversity is a major concern for dialogue generation. Same as in (Li et al., 2016a), we measure the diversity by the ratio of distinct unigrams ( Dist-1 ) and bigrams ( Dist-2 ) in all generated responses. As the ratio itself ignores the frequency distribution of n-grams, we further calculate the entropy value for the empirical distribution of n-grams (Zhang et al., 2018b). A larger entropy indicates more diverse distributions. We report the entropy of four-grams ( Ent-4 ) in Table 3. Among models trained only on the conversational corpus, the standard seq2seq performed worst as expected. All different variants improved the diversity more or less. Nucleus sampling and CVAE generated most diverse responses, especially Nucleus who wins on 6 out of the 8 metrics. By incorporating the non-conversational corpus, the diversity of generated responses improves dramatically. The retrieval-based system and our model perform best, in most cases even better than human references. This can happen as we enrich the response generation with external resources. The diversity would be more than the original conversational corpus. Weighted-average and multi-task models are relatively worse, though still greatly outperforming models trained only on the conversational corpus. We can also observe that our model improves over standard seq2seq only a bit after one iteration. As more iterations are added, the diversity improves gradually. Relevance Measuring the context-response relevance automatically is tricky in our case. The typical way of using scores from forward or backward models as in Li and Jurafsky (2017) is not suitable as our model borrowed information from extra resources. The generated responses are out-of-scope Metrics Weibo Douban Model BLEU-2 Dist-1 Dist-2 Ent-4 Adver BLEU-2 Dist-1 Dist-2 Ent-4 Adver STANDARD 0.0165 0.018 0.050 5.04 0.30 0.0285 0.071 0.206 7.55 0.19 MMI 0.0161 0.025 0.069 5.98 0.42 0.0263 0.143 0.363 7.60 0.31 DIVERSE 0.0175 0.019 0.054 6.20 0.38 0.0298 0.130 0.358 7.51 0.25 NUCLEUS 0.0183 0.027 0.074 7.41 0.43 0.0312 0.141 0.402 7.93 0.30 CVAE 0.0171 0.023 0.061 6.63 0.36 0.0287 0.169 0.496 7.80 0.29 RETRIEVAL 0.0142 0.198 0.492 12.5 0.13 0.0276 0.203 0.510 13.3 0.17 WEIGHTED 0.0152 0.091 0.316 9.26 0.22 0.0188 0.172 0.407 8.73 0.14 MULTI 0.0142 0.128 0.348 8.98 0.27 0.0110 0.190 0.389 8.26 0.16 BT (ITER =1) 0.0180 0.046 0.171 7.64 0.19 0.0274 0.106 0.313 8.16 0.15 BT (ITER =4) 0.0176 0.175 0.487 11.2 0.35 0.0269 0.207 0.502 11.0 0.25 HUMAN -0.171 0.452 9.23 0.88 -0.209 0.514 11.3 0.85 Table 3: Automatic evaluation on Weibo and Douban datasets. for the seq2seq model trained on only on the conversational corpus and thus would be assigned very low scores. Apart from the BLEU-2 score, we further evaluate the relevance by leveraging an adversarial discriminator (Li et al., 2017). As has been shown in previous research, discriminative models are generally less biased to high-frequent utterances and more robust against their generative counterparts (Lu et al., 2017; Luo et al., 2018). The discriminator is trained on the parallel conversational corpus distinguish correct responses from randomly sampled ones. We encode the context and response separately with two different LSTM neural networks and output a binary signal indicating relevant or not 11 . The relevance score is defined as the success rate that the model fools the adversarial classifier into believing its generations ( Adver in Table 3). The retrieval-based model, who generates the most diverse generations, achieve the lowest score as for relevance with context. The restriction that it can only select from a set of fixed utterances do affect the relevance a lot 12 . Note that the discriminator is also trained on the same bilateral conversational corpus, putting our model into a naturally disadvantageous place due to the incorporation of out-of-scope non-conversational text. Nonetheless, our model still achieves competitive relevance score even compared with models trained only on the conversational corpus. This suggests our model does learn the proper patterns in human conversations instead of randomly synthesizing diverse 11 In our experiment, the discriminator performs reasonably well in the 4 scenarios outlined in Li et al. (2017) and thus can be considered as a fair evaluation metric. 12 The fact that we only rank on 200 most similar utterances might also affect. We tried increasing the size to 1,000 but observe no tangible improvement. The candidate size required for a decent relevance score can be unbearably large. Apart from automatic evaluations, we also employed crowdsourced judges to evaluate the quality of generations for 500 contexts of each dataset. We focus on evaluating the generated responses regarding the (1) relevance: if they coincide with the context ( Rel ), (2) interestingness: if they are interesting for people to continue the conversation ( Inter ) and (3) fluency: whether they are fluent by grammar ( Flu ) 13 . Each sample gets one point if judged as yes and zero otherwise. Each pair is judged by three participants and the score supported by most people is adopted. The averaged scores are summarized in Table 4. We compare the standard seq2seq model, nucleus sampling which performs best among all seq2seq variants, and the four approaches leveraging the non-conversational text. All models perform decently well as for fluency except the weighted average one. The scores for diversity and relevance generally correlate well with the automatic evaluations. Overall the back-translation model are competitive with respect to fluency and relevance, while generating much more interesting responses to human evaluators. It also significantly outperforms the other three baseline 13 We do not evaluate the retrieval-based model for the fluency score as the retrieved utterances are fluent by construct. Effect of Iterative Training To show the importance of the iterative training paradigm, we visualize the change of the validation loss in Figure 2 14 . The forward validation loss is computed as the perplexity of the forward seq2seq on the pseudo context-response pairs obtained from the backward model, vice versa for backward loss. It approximately quantifies the KL divergence between them two (Kim and Rush, 2016; Cotterell and Kreutzer, 2018). As the iteration goes, the knowledge from the backward model is gradually distilled into the forward model. The divergence between them reaches the lowest point at iteration 4, where we stop our model. Table 5 further displays examples for different iterations. Iteration 0 generates mostly generic responses. Iteration 1 starts to become more diverse but still struggle with fluency and relevance. In the final iteration, it can learn to incorporate novel topics from the non-conversational text yet maintaining the relevance with context. Diversity of Generation We find the back translation model can generate both semantically and syntactically novel responses. Some examples are shown in Table 6. To find semantically novel responses, we segment them into phrases and find those containing novel phrases that do not exist on the conversational corpus. As in the first example of Table 6, the word    only exists in the non-conversational corpus. The model successfully learnt its semantic meaning and adopt it to generate novel responses. It is also common that the model learns frequent syntax structures from the non-conversational corpus. In the second example, it learnt the pattern of To ... is easy, to ... is hard, which appeared frequently in the non-conversational corpus, and utilized it to produce novel responses with the same structure. Note that both generations from the BT model never appear exactly in the non-conversational corpus . It must generate them by correctly understanding the meaning of the phrase components instead of memorizing the utterances verbally. We propose a novel way of diversifying dialogue generation by leveraging non-conversational text. To do so, we collect a large-scale corpus from forum comments, idioms and book snippets. By training the model through iterative back translation, it is able to significantly improve the diversity of generated responses both semantically and syntactically. We compare it with several strong baselines and find it achieved the best overall performance. The model can be potentially improved by filter-ing the corpus according to different domains, or augmenting with a retrieve-and-rewrite mechanism, which we leave for future work. We thank anonymous reviewers for valuable comments. Xiaoyu Shen is supported by IMPRS-CS fellowship. The work is partially funded by DFG collaborative research center SFB 1102.
abstain objective objective abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain method method method objective abstain abstain abstain method result other other other other other other other other other other objective other method abstain other other method abstain abstain method abstain method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain other abstain other abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain result method abstain method method method abstain method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method abstain result objective other other other