Emotion-controllable response generation is an attractive and valuable task that aims to make open-domain conversations more empathetic and engaging. Existing methods mainly enhance the emotion expression by adding regularization terms to standard cross-entropy loss and thus influence the training process. However, due to the lack of further consideration of content consistency, the common problem of response generation tasks, safe response, is intensified. Besides, query emotions that can help model the relationship between query and response are simply ignored in previous models, which would further hurt the coherence. To alleviate these problems, we propose a novel framework named Curriculum Dual Learning (CDL) which extends the emotion-controllable response generation to a dual task to generate emotional responses and emotional queries alternatively. CDL utilizes two rewards focusing on emotion and content to improve the duality. Additionally, it applies curriculum learning to gradually generate high-quality responses based on the difficulties of expressing various emotions. Experimental results show that CDL significantly outperforms the baselines in terms of coherence, diversity, and relation to emotion factors. Infusing emotions into dialogue systems can make conversational agents more human-like and ben-efit the interaction between human and machine (Prendinger and Ishizuka, 2005; Prendinger et al., 2005; Partala and Surakka, 2004). In some real-life scenarios, we need to customize and control the agent's emotion so that the agent can express a specific one. For example, in psychological counseling, the agent is supposed to express sadness to  Yang Feng is the corresponding author. show the sympathy and also convey happiness to cheer the patient up. 1 q It is very pleasant to have a cup of black tea with sugar on a cold day. (Happy) r 1 [Neural] It starts to cool down today. r 2 [Like] I will try, thanks for your advice. r 3 [Sad] I am frozen to death ... r 4 [Disgust] Winner is the worst season. r 5 [Angry] You know nothing! r 6 [Happy] I really like to drink black tea. 2 q So pets live better than humans now... (Sad) r 1 [Disgust] You are so bad. r 2 [Happy] Haha, you too. 3 q We should study hard. (Neural) r [Disgust] You are so bad. 4 q Happy birthday, Xinxin. May you be more beautiful, find a good person and get married soon! (Happy) r [Happy] Haha, you too. Recently, a framework called emotional chatting machine (ECM) (Zhou et al., 2018a) was proposed to address the emotion factor in a controlled manner, which focuses on generating a response with a specific emotion (Example 1 in Table 1). In the research field of emotion-controllable response generation, ECM and its successive methods (Colombo et al., 2019; Song et al., 2019) mainly represent the given emotion category as a vector and add it to the decoding steps to influence the procedure of response generation, which would aggravate the safe response problem. For the response generation task, safe response is notorious, as the model tends to produce some generic but meaningless responses, like Thank you, I don't know, Yes, etc. Due to the constraint of emotion factors, the scale of proper responses shrinks, and the model is more likely to map any query to a frequently-occurring response in that emotion category. That is, given Disgust, the response would be You are so bad in general, while given Happy, it would be Haha, you too (Example 2 to 4 in Table 1). Intuitively, for a good pair of query and response, they should be in a tight relationship and have equal qualities. Then, both the query-to-response mapping and response-to-query mapping would be easier and more natural. On the contrary, it is hard for a safe response to reach the original query through back-generation, neither on the content level nor the emotion level. At the same time, the difficulties of producing various emotions are different, especially in a noisy and uneven-quality dataset. Therefore, we can evaluate the response based on the feedback from the backward process to improve the coherence (Zhang et al., 2018; Cui et al., 2019; Luo et al., 2019b) and try to learn from easy to hard data to generate appropriate and emotion-rich responses. In this paper, we propose a new framework for emotion-controllable response generation named Curriculum Dual Learning (CDL). We take the learning of response and query generation with emotions as a dual task, and use the duality to model the mutual relation between them. The forward and backward models are trained alternatively via reinforcement learning (RL). Rewards designed here aim to encourage both emotion expression and content consistency. Specifically, emotion expression can be either explicit (embodied in some obvious emotion words) or implicit (reflected by the organization of the entire sentence). For example, I am happy to meet her again is explicit with the word happy, while It seems like I have eaten the honey is implicit, but the happiness can be felt when we consider the sentence as a whole. Based on these features, we use the accuracy of emotion classification of sentences and the proportion of emotion words as feedbacks for explicit and implicit emotions, respectively. For content consistency, we apply the reconstruction probability as the measurement of coherence (Sec-tion 3.1). Furthermore, in order to better utilize samples of multiple emotions from the noisy and uneven-quality dataset, we incorporate the curriculum learning (Section 3.2) into our dual learning framework (Section 3.3). Experimental results on both automatic and human evaluations show that for a given query and an emotion category, our CDL can successfully express desired emotion as well as keep the response informative and coherent to the query. For emotion-controllable response generation, given a query q and an emotion category e r , the goal is to generate a response r (cid:48) that is not only meaningful, but also in accordance with the desired emotion. Emotional Chatting Machine (ECM) (Zhou et al., 2018a) addresses the emotion factor using three new mechanisms: Emotion Category Embedding, Internal Memory, and External Memory. Specifically, 1) Emotion Category Embedding models the high-level abstraction of emotion expression by embedding emotion categories, and concatenates corresponding embedding to the input at each decoding step. 2) Internal Memory captures the change of implicit internal emotion states with read and write gates, 3) External Memory applies an external emotion vocabulary to express emotion explicitly, and finally assigns different generation probabilities to emotion and generic words. The loss function on one training sample ( q , r ) ( q = q 1 , q 2 , ..., q n , r = r 1 , r 2 , ..., r m ) is defined as:  m (cid:88) t =1 p t log ( o t )  m (cid:88) t =1 q t log (  t )+ || M Ie,m || , (1) where o t and p t are the predicted token distribution and gold distribution,  t is the probability of choosing an emotion word or a generic word, q t  { 0 , 1 } is the true choice between them in r , and M Ie,m is the internal emotion state at the last step m . The first term is the cross-entropy loss, the second one is used to supervise the probability of selecting an emotion or generic word, and the last one is used to ensure that the internal emotion state has been expressed completely once the generation is finished. Please refer to the original paper for more details. Since our CDL method is a combination of dual learning (DL) and curriculum learning (CL), we first present the main components of DL, including states, actions, policy and reward, then introduce the plausibility of curriculum learning. Finally, we describe the training algorithm of CDL. The architecture of DL is illustrated in Figure 1. Both the forward model M f and the backward model M b are ECMs with independent parameters and are initialized according to the maximum likelihood estimation (MLE). CLS is a pre-trained classifier that calculates the score of implicit emotion expression. In general, M f generates a response r (cid:48) for a given query q and emotion category e r , and then obtains the reward R that consists of R e from CLS and R c from M b (red parts in Figure 1). Similarly, M b generates a query q (cid:48) for a given response r and emotion category e q , and obtains the reward R that consists of R e and R c from CLS and M f (blue parts in Figure 1). These two models are trained alternatively via reinforcement learning (RL). Specifically, an action is the dialogue response to generate. The action space is infinite since arbitrary-length sequences can be generated. A state is denoted by the query, which is further transformed to a vector representation by the encoder. A policy takes the form of a GRU encoder-decoder and is defined by its parameters. Following the work of Li et al. (2016c); Zhang et al. (2018), we use a stochastic representation of the policy, i.e., a probability distribution over actions given states. In order to encourage both content consistency and emotion expression, we introduce two rewards and use them to train M f and M b . The definition of the two rewards for model M f is introduced as follows 1 . Reward for emotion expression For implicit emotion expression, a straightforward method is to employ the pre-trained classifier CLS to evaluate the emotion category of the generated response r (cid:48) , and use the classification accuracy as the reward: R e 1 ( q , r (cid:48) ) = p ( e r | r (cid:48) ;  ) , (2) where  is the parameter of CLS , and it is fixed during training. where n ( w e r ) is the number of emotion words belong to category e r , and | r (cid:48) | is the length of r (cid:48) . Then, the emotion reward is defined as: R e ( q , r (cid:48) ) = R e 1 ( q , r (cid:48) ) + R e 2 ( q , r (cid:48) ) , (4) where  controls the relative importance of implicit and explicit rewards. Reward for content consistency If the response are coherent and related to the query, it will be easier to reproduce the query via back generation. Inspired by Zhang et al. (2018); Cui et al. (2019); Luo et al. (2019b), we measure the coherence by means of reconstructing q conditioned on r (cid:48) . Formally, the content consistency reward is defined as: R c ( q , r (cid:48) ) = p ( q | r (cid:48) , e q ;  ) , (5) where  is the parameter of backward model M b , and it is fixed during the training of M f . Overall reward We use the weighted sum of the above two rewards as the final reward: R ( q , r (cid:48) ) = R c ( q , r (cid:48) ) + R e ( q , r (cid:48) ) , (6) where  is a hyper-parameter that controls the tradeoff between R c ( q , r (cid:48) ) and R e ( q , r (cid:48) ) . Intuitively, learning from less noisy and even-quality dataset is simpler, but in this task, the data is inherently complicated as there are multiple emotions mixed in it. To better utilize the data, we integrate curriculum learning into the dual learning framework. The core of curriculum learning 1 Rewards for model M b can be computed in a similar way, where q (cid:48) , r , b and f replace r (cid:48) , q , f and b , respectively. Therefore, we omit them here for space limitation and brevity. (Bengio et al., 2009) is to design an evaluation for complexity, and to provide the model with easy samples first, then gradually increase the difficulty. The curriculum is arranged by sorting each sample in training set according to a specific ranking standard. Here, We reorder samples from easy, i.e., with high accuracy of emotion classification, to hard. We consider the classification accuracy after pretraining as an indicator of the learning order. Another intuitive way is to put emotionless samples (labelled as Neural) first and then emotional ones, however, it exhibits poor performance in our experiments. At training step t , a batch of training samples is obtained from the top f ( t ) portions of the entire sorted training samples. Following Pla-tanios et al. (2019) and Cai et al. (2020), we define the function f ( t ) as: f ( t ) (cid:44) min(1 , (cid:114) t (1  c 20 ) T + c 2 0 ) , (7) where c 20 is set to 0.01, which means that the model starts training using the 1% easiest training samples, and T is a hyper-parameter that represents the duration of curriculum learning (curriculum length). At the early stage of the training process, the model learns from the samples in the easy part of the curriculum, where there is only one emotion category. As the advance of the curriculum, the difficulty gradually increases, as complex training samples from more different categories appear. After training T batches, training sample of each batch is drawn from the whole training set, which is the same as the conventional training procedure. Optimization We use the policy gradient method (Williams, 1992) to find parameters that lead to a larger expected reward. For the forward learning process, the expected reward of the generated response r (cid:48) and its approximate gradient are defined as: J (  ) = E [ R ( q , r (cid:48) ) ] , (8)   J (  ) (cid:39) R (cid:48) ( q , r (cid:48) )    log( p  ( r (cid:48) | q , e r )) , (9) where  is the parameter of forward model M f , R (cid:48) ( q , r (cid:48) ) = R ( q , r (cid:48) )  b f , and b f is the baseline value from the greedy search decoding method for M f , which is used to reduce the variance of the estimation (Zaremba and Sutskever, 2015; Paulus et al., 2017). Analogously, for the backward learning process, the expected reward of the generated query q (cid:48) and corresponding approximate gradient are defined as: J (  ) = E [ R ( r , q (cid:48) ) ] , (10)   J (  ) (cid:39) R (cid:48) ( r , q (cid:48) )    log( p  ( q (cid:48) | r , e q )) , (11) where  is the parameter of backward model M b , R (cid:48) ( r , q (cid:48) ) = R ( r , q (cid:48) )  b b , and b b is the baseline value from the greedy search decoding method for M b . corresponding emotion labels e q i and e r i Output: M f and M b 1: Pre-train M f and M b with ( q i , r i , e r i ) and ( r i , q i , e q i ) , respectively, based on Eq. 1 2: Pre-train CLS with ( q i , e q i ) and ( r i , e r i ) 3: Sort training samples according to the ranking standard in Section 3.2 for both forward and backward learning process to get D f and D b 4: for training step t = 1 , ..., T do 5: (cid:46) Train M f 6: Sample a batch B f t in D f based on Eq. 7 7: Sample ( q, r, e r ) from B f t 8: Generate response r (cid:48) via M f 9: Compute reward R based on Eq. 6 10: Update  using R based on Eq. 9 11: Teacher Forcing: Update  with ( q, r, e r ) 12: (cid:46) Train M b 13: Sample a batch B b t in D b based on Eq. 7 14: Sample ( r, q, e q ) from B b t 15: Generate response q (cid:48) via M b 16: Compute reward R based on Eq. 6 17: Update  using R based on Eq. 11 18: Teacher Forcing: Update  with ( r, q, e q ) 19: end for Teacher Forcing When M f and M b are trained with only the rewards from the dual tasks, the training process would easily collapse as it may find an unexpected way to achieve a high reward but fail to guarantee the fluency or readability of the generated text (Ranzato et al., 2015; Pasunuru and Bansal, 2018; Luo et al., 2019b). To stabilize the training process, after each update according to Eq. 9 or 11, M f or M b is exposed to real query-response pairs and is trained via MLE, which is also known as Teacher Forcing (Li et al., 2017; Lamb et al., 2016). M f , M b and CLS with query-response pairs and emotion labels in the training set. After the pretraining phase, we sort samples in the training set following the ranking standard in Section 3.2. For forward learning process, the ranking is based on responses, while for backward learning process, it is based on queries. Then, we can get two sorted training set D f and D b for each direction. Finally, M f and M b are optimized with rewards and the regularization of Teacher Forcing, alternatively. In this section, we conduct experiments to evaluate our proposed method. We first introduce some empirical settings, including dataset, hyper-parameters, baselines, and evaluation measures. Then we illustrate our results under both automatic and human evaluations. Finally, we give out some cases generated by different models and do further analyses over our method. We apply our method on the corpus of NLPCC 2017 Emotional Conversation Generation Challenge 2 , namely NLPCC2017 Dataset, which is an extension version of the dataset collected by Zhou et al. (2018a). The provided dataset is already segmented into Chinese words. There are over 1 million query-response pairs, in which both the query and response are labelled with one emotion tag among Happy, Angry, Disgust, Sad, Like and Neutral. The dataset has been tokenized into words. We randomly split the whole dataset into training/validation/test set with the number of 1,105,487/11,720/2,000. The detailed statistics of training set are shown in Table 2. Training Emotion Query Response Happy 120,358 197,528 Angry 79,611 138,198 Disgust 184,427 197,428 Sad 128,482 179,215 Like 257,471 197,565 Neutral 335,138 195,553 1,105,487 Validation 11,720 Test 2,000 Table 2: Statistics of the NLPCC2017 Dataset. The settings of both M f and M b follow the default implementation details of original ECM paper (Zhou et al., 2018a), where the encoder and decoder have 2-layer GRU structures with 256 hidden cells for each layer, the embedding size of words and emotion categories are set to 100, and the vocabulary size is limited to 40,000. The minimum and maximum sentence length is set to 3 and 30, respectively. We train a TextCNN-based classifier (Kim, 2014) and the classification accuracy reaches 65.6% on the test set, which has the similar performance with those used by (Zhou et al., 2018a) and (Song et al., 2019). Before curriculum dual learning, model M f and M b are pre-trained 10 epochs via MLE. The optimizer is Adam (Kingma and Ba, 2015) with 0.05 initial learning rate for pre-training and 10  5 for curriculum dual learning. The batch size is set to 64. in Eq. 4 is 0.5,  in Eq. 6 is 1 and T in Eq. 7 is 100k. During curriculum dual learning, training runs until the performance on validation set does not improve. We compare our approach with four representative baselines: (1) S2S-Attn : The Seq2Seq model with attention mechanism as in Shang et al. (2015). (2) EmoEmb : A Seq2Seq variant which takes the embedding of emotion categories as additional input at each decoding position (Ficler and Goldberg, 2017; Li et al., 2016b). (3) EmoDS : An emotional dialogue system with lexicon-based attention and a word-based classifier (Song et al., 2019). (4) ECM : Emotional Chatting Machine proposed by Zhou et al. (2018a). Additionally, we also conduct ablation study to better analyze our method as follows: (5) CDL-emo : CDL with emotion reward only; (6) CDL-con : CDL with content reward only, which is similar to the work of Zhang et al. (2018); (7) CDL-DL : CDL with both rewards but without curriculum learning. To better evaluate our results, we use both quantitative metrics and human judgements in our experiments. For automatic evaluation, we mainly choose four kinds of metrics: 1) Embedding scores (Average, Greedy, Extrema and Coherence) 3 (Liu et al., 2016; Xu et al., 2018); 2) BLEU scores (Papineni et al., 2002) in 0 to 1 scale; 3) Dist-1, Dist-2 (Li et al., 2016a) and 4) Emotion-acc, Emotion-word (Zhou et al., 2018a; Song et al., 2019). Embedding scores and BLEU scores are used to measure the quality of generated responses in terms of content relevance. Whereas, Dist-1 and Dist-2 are used to evaluate the diversity of responses 4 . Emotion-acc and Emotion-word are utilized to test the emotion expression. Specifically, Emo-acc is the agreement between the ground truth labels and the predicted labels through the TextCNN classifier trained before. Emo-word is the percentage of the generated responses that contain the corresponding emotion words. Since there are no multi-emotion ground truths in the test set, we only calculate the metrics between the ground truth, labelled emotion e , and the generated response given also label e for fair comparison. Inspired by Zhou et al. (2018a); Song et al. (2019), a human evaluation is conducted to better analyze the quality of generated responses. First, we randomly sample 200 queries from the test set. For 3 We use the pre-trained word embeddings based on Sina Weibo data from https://github.com/Embedding/ Chinese-Word-Vectors . 4 We employ a popular NLG evaluation project available at https://github.com/Maluuba/nlg-eval for automatic evaluation. each method except S2S-Attn, they generate six responses for six emotion categories, while S2S-Attn generates top 6 responses from beam search decoding for each query. Then, we send the triples of (query, response, emotion) to three human annotators without order, and require them to evaluate each response on both content level and emotion level independently. Content and emotion are measured by a 3-scale rating (0, 1, 2) and a 2-scale rating (0, 1), respectively. Evaluation from the content level assesses whether a response is fluent, coherent and meaningful for the query, and evaluation from the emotion level decides if a response reveals the desired emotion. The automatic results are shown in Table 3. The top part is the results of all baseline models, and we can see that CDL outperforms the other methods on all metrics (t-test, p -value < 0.05). The improvements of CDL on Coherence , Emotion-acc and Emotion-word are significant, indicating that it can enhance content consistency and emotion expression simultaneously. EmoDS and ECM have similar performance, as both of them use the forward method to pay more attention on the emotion Query Method Response Translated Response S2S-Attn factor. S2S-Attn can only generate fluent responses based on semantic mapping, but fail to express diverse responses. The bottom part of Table 3 shows the results of our ablation study. Comparisons among CDL-emo, CDL-con and CDL show the effectiveness of the combined reward for both emotion expression and content consistency. In addition, we can find that with the support of curriculum learning, CDL can achieve better results than CDL-DL. The results are shown in Table 4. CDL obtains the best performance (t-test, p -value < 0.05) on both emotion expression (0.582) and content coherence (1.286). As we can see, there is no obvious difference between EmoDS and ECM. Due to the insuf-ficient training data of Anger (79,611 in queries and 138,198 in responses), S2S-Attn achieves the best content score for it, which is similar to the results of Zhou et al. (2018a). To better evaluate the overall quality of the generated responses, we present results in Table 6 by considering content and emotion scores simultaneously. 32.5% of the responses generated by CDL are annotated with Emotion score 2 and Content score 1, which shows that CDL is better at producing coherent as well as emotion-rich responses. Agreements to measure the consistency among three annotators are calculated with the Fleiss' kappa (Fleiss and Cohen, 1973). Fleiss' kappa for content and emotion is 0.497 and 0.825, indicating Moderate agreement and Substantial agreement, respectively. Table 5 shows the examples generated by S2S-Attn, ECM and CDL. As can be seen from it, for a given post, there are multiple emotion categories that are appropriate for its response in the conversation. S2S-Attn generates a response with a random emotion, while ECM and CDL can utilize the specific emotion label. Compared with ECM, CDL can generate both coherent and informative responses with any desired emotion. In addition, the emotion can be expressed in either explicit or implicit manner. For example,   /   /   /    (You do not understand life at all!) express anger when we read this sentence as a whole, while    (beauti-ful) or   (happy) are strong emotion words to represent Like or Happy. Here, we conduct a further analysis to show some characteristics of this task and the effect of CDL. Emotion lexicon size and classification accuracy after pre-training of each category ( N (correct prediction)  category size) are listed in Table 7. We can see that the classification accuracy is not totally related to the emotion lexicon size, indicating the emotion expression is partially implicit or explicit. To better illustrate the learning efficiency of CDL, we plot the changes of Emotion-acc on the validation set. As shown in Figure 2, CDL accelerates the learning effectively and consistently outperforms CDL-DL. Responses generated by traditional open-domain dialogue systems are usually safe and generic. To produce diverse and informative responses, researchers tried to either import latent variables for model construction (Zhao et al., 2017; Serban et al., 2017; Shen et al., 2019) or utilize some extra knowledge, e.g., sentence types, personas, emotions, documents and knowledge triples/graphs (Ke et al., 2018; Li et al., 2016b; Zhou et al., 2018a; Meng et al., 2019; Zhou et al., 2018b; Niu et al., 2019). In this paper, we mainly touch on two branches of research: emotional response generation and dual learning in NLP. Early studies have proven that dialogue systems with proper emotional expressions and reactions can directly improve user satisfaction (Prendinger and Ishizuka, 2005; Prendinger et al., 2005) and contribute to effective users' performance (Partala and Surakka, 2004). Polzin and Waibel (2000) and Polzin and Waibel (2000) apply rule-based methods to choose emotional responses from a conversation corpus, but those rules are hard to extend to large corpora. With the advent of deep learning, some researchers utilize neural networks to solve this problem (Ghosh et al., 2017; Hu et al., 2017; Zhou and Wang, 2018; Sun et al., 2018). Besides, the Valence, Arousal, and Dominance (VAD) lexicon (Warriner et al., 2013; Mohammad, 2018) is embedded to the sequence-to-sequence model (Sutskever et al., 2014) to provide extra affective information (Asghar et al., 2018; Zhong et al., 2019). Responses generated by above studies can simply continues the emotion of the query. To generate emotion-controllable responses, Zhou et al. (2018a) address the emotion factor in large-scale conversations, and propose ECM to generate responses based on different given emotions. After that, Colombo et al. (2019) augment ECM with VAD embeddings and modified the loss function and decoding procedure. Song et al. (2019) use lexicon-based attention and a word-based classifier to improve the ability of emotion expression. He et al. (2016) propose Dual Learning (DL) for machine translation first which consider the source to target language translation and target to source language translation as a dual task. After that, Tang et al. (2017) implement a dual framework for the question answering system. Both Zhang et al. (2018) and Cui et al. (2019) use similar idea in dialogue generation task to produce coherent but not safe responses, since they find that a more diverse and specific response usually has a higher probability of being transformed back to the given query. Luo et al. (2019b) and Luo et al. (2019a) exploit DL in unsupervised text style transfer to relieve the need of parallel data. The differences between our method and those in Section 5.1 and Section 5.2 are: (1) We consider the emotion expression and content consistency simultaneously via a DL method. (2) Instead of regarding the query as an emotionless sentence, we utilize the emotion of query, which can help model the emotion shifting and coherence to improve the quality of response. (3) To better model the changes in emotion and content between the query and response, we combine the DL method with curriculum learning, which is known to improve the effectiveness and generalization. In this paper, we propose a new framework Curriculum Dual Learning (CDL) for generating emotional responses in a controlled manner. Since existing methods in this field only focus on the emotion expression of target label but fail to consider the emotion of queries, the safe response problem deteriorates and hurts the content consistency. CDL utilizes two kinds of rewards to enhance emotion and content simultaneously via dual learning. Besides, with the support of curriculum learning, it can be more efficient. Experimental results show that CDL can generate fluent, coherent, informative as well as emotional responses. This work was supported by National Key R&D Program of China (NO. 2017YFE0192900). We sincerely thank the anonymous reviewers for their helpful and valuable suggestions.
abstain abstain abstain abstain objective abstain abstain abstain abstain method abstain abstain objective objective objective objective objective objective objective method objective objective objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain result objective method abstain abstain abstain method method method method result other other other other other other other objective other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain other abstain abstain abstain method abstain method abstain method abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain other other abstain other abstain abstain abstain abstain abstain other abstain abstain abstain method abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain other abstain abstain abstain abstain abstain abstain abstain abstain other other abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain result abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain other other abstain other other other other other other other other other other other other method abstain abstain objective abstain abstain abstain abstain other other