Human conversations naturally evolve around related concepts and scatter to multi-hop concepts. This paper presents a new conversation generation model, ConceptFlow, which leverages commonsense knowledge graphs to explicitly model conversation flows. By grounding conversations to the concept space, ConceptFlow represents the potential conversation flow as traverses in the concept space along commonsense relations. The traverse is guided by graph attentions in the concept graph, moving towards more meaningful directions in the concept space, in order to generate more semantic and informative responses. Experiments on Reddit conversations demonstrate ConceptFlow's effectiveness over previous knowledge-aware conversation models and GPT-2 based models while using 70% fewer parameters, confirming the advantage of explicit modeling conversation structures. All source codes of this work are available at https://github.com/ thunlp/ConceptFlow . The rapid advancements of language modeling and natural language generation (NLG) techniques have enabled fully data-driven conversation models, which directly generate natural language responses for conversations (Shang et al., 2015; Vinyals and Le, 2015; Li et al., 2016b). However, it is a common problem that the generation models may degenerate dull and repetitive contents (Holtzman et al., 2019; Welleck et al., 2019), which, in conversation assistants, leads to off-topic and useless responses. (Tang et al., 2019; Zhang et al., 2018; Gao et al., 2019). Conversations often develop around Knowledge. A promising way to address the degeneration probIndicates equal contribution. lem is to ground conversations with external knowledge (Xing et al., 2017), such as open-domain knowledge graph (Ghazvininejad et al., 2018), commonsense knowledge base (Zhou et al., 2018a), or background documents (Zhou et al., 2018b). Recent research leverages such external knowledge by using them to ground conversations, integrating them as additional representations, and then generating responses conditioned on both the texts and the grounded semantics (Ghazvininejad et al., 2018; Zhou et al., 2018a,b). Integrating external knowledge as extra semantic representations and additional inputs to the conversation model effectively improves the quality of generated responses (Ghazvininejad et al., 2018; Logan et al., 2019; Zhou et al., 2018a). Nevertheless, some research on discourse development suggests that human conversations are not still: People chat around a number of related concepts, and shift their focus from one concept to others. Grosz and Sidner (1986) models such concept shift by breaking discourse into several segments, and demonstrating different concepts, such as objects and properties, are needed to interpret different discourse segments. Attentional state is then introduced to represent the concept shift corresponding to each discourse segment. Fang et al. (2018) shows that people may switch dialog topics entirely in a conversation. Restricting the utilization of knowledge only to those directly appear in the conversation, effective as they are, does not reach the full potential of knowledge in modeling human conversations. To model the concept shift in human conversations, this work presents ConceptFlow ( Con versation generation with Con cept Flow ), which leverages commonsense knowledge graphs to model the conversation flow in the explicit concept space. For example, as shown in Figure 1, the concepts of a conversation from Reddit evolves from chat and future, to adjacent concept talk, and also hops to distant concept dream along the commonsense relationsa typical involvement in natural conversations. To better capture this conversation structure, ConceptFlow explicitly models the conversations as traverses in commonsense knowledge graphs: it starts from the grounded concepts, e.g., chat and future, and generates more meaningful conversations by hopping along the commonsense relations to related concepts, e.g., talk and dream. The traverses in the concept graph are guided by graph attention mechanisms, which derives from graph neural networks to attend on more appropriate concepts. ConceptFlow learns to model the conversation development along more meaningful relations in the commonsense knowledge graph. As a result, the model is able to grow the grounded concepts by hopping from the conversation utterances, along the commonsense relations, to distant but meaningful concepts; this guides the model to generate more informative and on-topic responses. Modeling commonsense knowledge as concept flows, is both a good practice on improving response diversity by scattering current conversation focuses to other concepts (Chen et al., 2017), and an implementation solution of the attentional state mentioned above (Grosz and Sidner, 1986). Our experiments on a Reddit conversation dataset with a commonsense knowledge graph, ConceptNet (Speer et al., 2017), demonstrate the effectiveness of ConceptFlow. In both automatic and human evaluations, ConceptFlow significantly outperforms various seq2seq based generation models (Sutskever et al., 2014), as well as previous methods that also leverage commonsense knowledge graphs, but as static memories (Zhou et al., 2018a; Ghazvininejad et al., 2018; Zhu et al., 2017). Notably, ConceptFlow also outperforms two fine-tuned GPT-2 systems (Radford et al., 2019), while using 70% fewer parameters. Explicitly modeling conversation structure provides better parameter efficiency. We also provide extensive analyses and case studies to investigate the advantage of modeling conversation flow in the concept space. Our analyses show that many Reddit conversations are naturally aligned with the paths in the commonsense knowledge graph; incorporating distant concepts significantly improves the quality of generated responses with more on-topic semantic information added. Our analyses further confirm the effectiveness of our graph attention mechanism in selecting useful concepts, and ConceptFlow's ability in leveraging them to generate more relevant, informative, and less repetitive responses. Sequence-to-sequence models, e.g., Sutskever et al. (2014), have been widely used for natural language generation (NLG), and to build conversation systems (Shang et al., 2015; Vinyals and Le, 2015; Li et al., 2016b; Wu et al., 2019). Recently, pre-trained language models, such as ELMO (Devlin et al., 2019), UniLM (Dong et al., 2019) and GPT-2 (Radford et al., 2018), further boost the NLG performance with large scale pretraining. Nevertheless, the degenerating of irrelevant, off-topic, and non-useful responses is still one of the main challenges in conversational generation (Rosset et al., 2020; Tang et al., 2019; Zhang et al., 2018; Gao et al., 2019). Recent work focuses on improving conversation generation with external knowledge, for example, incorporating additional texts (Ghazvininejad et al., 2018; Vougiouklis et al., 2016; Xu et al., 2017; Long et al., 2017), or knowledge graphs (Long et al., 2017; Ghazvininejad et al., 2018). They have shown external knowledge effectively improves conversation response generation. The structured knowledge graphs include rich semantics represented via entities and relations (Hayashi et al., 2019). Lots of previous studies focus on task-targeted dialog systems based on domain-specific knowledge bases (Xu et al., 2017; Zhu et al., 2017; Gu et al., 2016). To generate responses with a large-scale knowledge base, Zhou et al. (2018a) and Liu et al. (2018) utilize graph attention and knowledge diffusion to select knowledge semantics for utterance understanding and response generation. Moon et al. (2019) focuses on the task of entity selection, and takes advantage of positive entities that appear in the golden response. Different from previous research, ConceptFlow models the conversation flow explicitly with the commonsense knowledge graph and presents a novel attention mechanism on all concepts to guide the conversation flow in the latent concept space. This section presents our Con versation generation model with latent Con cept Flow (ConceptFlow). Our model grounds the conversation in the concept graph and traverses to distant concepts along commonsense relations to generate responses. Given a user utterance X = { x 1 , ..., x m } with m words, conversation generation models often use an encoder-decoder architecture to generate a response Y = { y 1 , ..., y n } . The encoder represents the user utterance X as a representation set H = { (cid:126)h 1 , ...,(cid:126)h m } . This is often done by Gated Recurrent Units (GRU): (cid:126)h i = GRU ( (cid:126)h i  1 ,(cid:126)x i ) , (1) where the (cid:126)x i is the embedding of word x i . The decoder generates t -th word in the response according to the previous t  1 generated words y <t = { y 1 , ..., y t  1 } and the user utterance X : P ( Y | X ) = n (cid:89) t =1 P ( y t | y <t , X ) . (2) Then it minimizes the cross-entropy loss L and optimizes all parameters end-to-end: L = n (cid:88) t =1 CrossEntropy ( y  t , y t ) , (3) where y  t is the token from the golden response. The architecture of ConceptFlow is shown in Figure 2. ConceptFlow first constructs a concept graph G with central graph G central and outer graph G outer according to the distance (hops) from the grounded concepts (Sec. 3.2). Then ConceptFlow encodes both central and outer concept flows in central graph G central and outer graph G outer , using graph neural networks and concept embedding (Sec. 3.3). The decoder, presented in Section 3.4, leverages the encodings of concept flows and the utterance to generate words or concepts for responses. ConceptFlow constructs a concept graph G as the knowledge for each conversation. It starts from the grounded concepts (zero-hop concepts V 0 ), which appear in the conversation utterance and annotated by entity linking systems. Then, ConceptFlow grows zero-hop concepts V 0 with one-hop concepts V 1 and two-hop concepts V 2 . Concepts from V 0 and V 1 , as well as all relations between them, form the central concept graph G central , which is closely related to the current conversation topic. Concepts in V 1 and V 2 and their connections form the outer graph G outer . The constructed concept graph provides explicit semantics on how concepts related to commonsense knowledge. ConceptFlow utilizes it to model the conversation and guide the response generation. It starts from the user utterance, traversing through central graph G central , to outer graph G outer . This is modeled by encoding the central and outer concept flows according to the user utterance. Central Flow Encoding. The central concept graph G central is encoded by a graph neural network that propagates information from user utterance H to the central concept graph. Specifically, it encodes concept e i  G central to representation (cid:126)g e i : (cid:126)g e i = GNN ( (cid:126)e i , G central , H ) , (4) where (cid:126)e i is the concept embedding of e i . There is no restriction of which GNN model to use. We choose Sun et al. (2018)'s GNN (GraftNet), which shows strong effectiveness in encoding knowledge graphs. More details of GraftNet can be found in Appendix A.3. Outer Flow Encoding. The outer flow f e p , hopping from e p  V 1 to its connected two-hop concept e k , is encoded to (cid:126)f e p by an attention mechanism: (cid:126)f e p = (cid:88) e k  e k  [ (cid:126)e p  (cid:126)e k ] , (5) where (cid:126)e p and (cid:126)e k are embeddings for e p and e k , and are concatenated (  ). The attention  e k aggregates concept triple ( e p , r, e k ) to get (cid:126)f e p :  e k = softmax (( w r  (cid:126)r ) (cid:62)  tanh( w h  (cid:126)e p + w t  (cid:126)e k )) , (6) where (cid:126)r is the relation embedding between the concept e p and its neighbor concept e k . w r , w h and w t are trainable parameters. It provides an efficient attention specifically focusing on the relations for multi-hop concepts. To consider both user utterance and related information, the texts from the user utterance and the latent concept flows are incorporated by decoder using two components: 1) the context representation that combines their encodings (Sec. 3.4.1); 2) the conditioned generation of words and concepts from the context representations (Sec. 3.4.2). time decoding with the encodings of the utterance and the latent concept flow. Specifically, (cid:126)s t is calculated by updating the ( t  1 )-th step output representation (cid:126)s t  1 with the ( t  1 )-th step context representation (cid:126)c t  1 : (cid:126)s t = GRU ( (cid:126)s t  1 , [ (cid:126)c t  1  (cid:126)y t  1 ]) , (7) where (cid:126)y t  1 is the ( t  1 )-th step generated token y t  1 's embedding, and the context representation (cid:126)c t  1 concatenates the text-based representation (cid:126)c text t  1 and the concept-based representation (cid:126)c concept t  1 : (cid:126)c t  1 = FFN ([ (cid:126)c text t  1  (cid:126)c cpt t  1 ]) . and attentions  jt  1 on the utterance tokens: (cid:126)c cpt t  1 =   (cid:88) e i  G central  e i t  1  (cid:126)g e i      (cid:88) f ep  G outer  ft  1  (cid:126)f e p   . (11) The attention  e i t  1 weights over central concept representations:  e i t  1 = softmax ( (cid:126)s t  1  (cid:126)g e i ) , (12) and the attention  ft  1 weights over outer flow representations:  ft  1 = softmax ( (cid:126)s t  1  (cid:126)f e p ) . (13) 3.4.2 Generating Tokens The t -th time output representation (cid:126)s t (Eq. 7) includes information from both the utterance text, the concepts with different hop steps, and the attentions upon them. The decoder leverages (cid:126)s t to generate the t -th token to form more informative responses. It first uses a gate   to control the generation by choosing words (   = 0 ), central concepts ( V 0 , 1 ,   = 1 ) and outer concept set ( V 2 ,   = 2 ):   = argmax  { 0 , 1 , 2 } ( FFN  ( (cid:126)s t )) , (14) The generation probabilities of word w , central concept e i , and outer concepts e k are calculated over the word vocabulary, central concept set V 0 , 1 , and outer concept set V 2 : y t    softmax ( (cid:126)s t  (cid:126)w ) ,   = 0 softmax ( (cid:126)s t  (cid:126)g e i ) ,   = 1 softmax ( (cid:126)s t  (cid:126)e k ) ,   = 2 , (15) where (cid:126)w is the word embedding for word w , (cid:126)g e i is the central concept representation for concept e i and (cid:126)e k is the two-hop concept e k 's embedding. The training and prediction of ConceptFlow are conducted following standard conditional language models, i.e. using Eq. 15 in place of Eq. 2 and training it by the Cross-Entropy loss (Eq. 3). Only ground truth responses are used in training and no additional annotation is required. This section describes the dataset, evaluation metrics, baselines, and implementation details of our experiments. Dataset. All experiments use the multi-hop extended conversation dataset based on a previous dataset which collects single-round dialogs from Reddit (Zhou et al., 2018a). Our dataset contains 3,384,185 training pairs and 10,000 test pairs. Preprocessed ConceptNet (Speer et al., 2017) is used as the knowledge graph, which contains 120,850 triples, 21,471 concepts and 44 relation types. Evaluation Metrics. A wide range of evaluation metrics are used to evaluate the quality of generated responses: PPL (Serban et al., 2016), Bleu (Papineni et al., 2002), Nist (Doddington, 2002), ROUGE (Lin, 2004) and Meteor (Lavie and Agarwal, 2007) are used for relevance and repetitiveness; Dist-1, Dist-2 and Ent-4 are used for diversity, which is same with the previous work (Li et al., 2016a; Zhang et al., 2018). The metrics above are evaluated using the implementation from Galley et al. (2018). Zhou et al. (2018a)'s concept PPL mainly focuses on concept grounded models and this metric is reported in Appendix A.1. The Precision, Recall, and F1 scores are used to evaluate the quality of learned latent concept flow in predicting the golden concepts which appear in ground truth responses. Baselines. The six baselines compared come from three groups: standard Seq2Seq, knowledge-enhanced ones, and fine-tuned GPT-2 systems. Seq2Seq (Sutskever et al., 2014) is the basic encoder-decoder for language generation. Knowledge-enhanced baselines include MemNet (Ghazvininejad et al., 2018), CopyNet (Zhu et al., 2017) and CCM (Zhou et al., 2018a). MemNet maintains a memory to store and read concepts. CopyNet copies concepts for the response generation. CCM (Zhou et al., 2018a) leverages a graph attention mechanism to model the central concepts. These models mainly focus on the grounded concepts. They do not explicitly model the conversation structures using multi-hop concepts. GPT-2 (Radford et al., 2019), the pre-trained model that achieves the state-of-the-art in lots of language generation tasks, is also compared in our experiments. We fine-tune the 124M GPT-2 in two ways: concatenate all conversations together and train it like a language model (GPT-2 lang ); extend the GPT-2 model with encode-decoder architecture and supervise with response data (GPT-2 conv ). Implement Details. The zero-hop concepts are initialized by matching the keywords in the post to concepts in ConceptNet, the same with CCM (Zhou et al., 2018a). Then zero-hop concepts are extended to their neighbors to form the central concept graph. The outer concepts contain a large amount of two-hop concepts with lots of noises. To reduce the computational cost, we first train ConceptFlow (se-lect) with 10% random training data, and use the learned graph attention to select top 100 two-hop concepts over the whole dataset. Then the standard train and test are conducted with the pruned graph. More details of this filtering step can be found in Appendix A.4. TransE (Bordes et al., 2013) embedding and Glove (Pennington et al., 2014) embedding are used to initialize the representation of concepts and words, respectively. Adam optimizer with the learning rate of 0.0001 is used to train the model. Five experiments are conducted to evaluate the generated responses from ConceptFlow and the effectiveness of the learned graph attention. This experiment evaluates the generation quality of ConceptFlow automatically and manually. Automatic Evaluation. The quality of generated responses is evaluated with different metrics from three aspects: relevance, diversity, and novelty. Table 1 and Table 2 show the results. In Table 1, all evaluation metrics calculate the relevance between the generated response and the Model Bleu-4 Nist-4 Rouge-1 Rouge-2 Rouge-L Meteor PPL Seq2Seq 0.0098 1.1069 0.1441 0.0189 0.1146 0.0611 48.79 MemNet 0.0112 1.1977 0.1523 0.0215 0.1213 0.0632 47.38 CopyNet 0.0106 1.0788 0.1472 0.0211 0.1153 0.0610 43.28 CCM 0.0084 0.9095 0.1538 0.0211 0.1245 0.0630 42.91 GPT-2 (lang) 0.0162 1.0844 0.1321 0.0117 0.1046 0.0637 29.08  GPT-2 (conv) 0.0124 1.1763 0.1514 0.0222 0.1212 0.0629 24.55  ConceptFlow 0.0246 1.8329 0.2280 0.0469 0.1888 0.0942 29.90 Table 1: Relevance Between Generated and Golden Responses. golden response. ConceptFlow outperforms all baseline models by large margins. The responses generated by ConceptFlow are more on-topic and match better with the ground truth responses. In Table 2, Dist-1, Dist-2, and Ent-4 measure the word diversity of generated responses and the rest of metrics measure the novelty by comparing the generated response with the user utterance. ConceptFlow has a good balance in generating novel and diverse responses. GPT-2's responses are more diverse, perhaps due to its sampling mechanism during decoding, but are less novel and on-topic compared to those from ConceptFlow. Human Evaluation. The human evaluation focuses on two aspects: appropriateness and informativeness. Both are important for conversation systems (Zhou et al., 2018a). Appropriateness evaluates if the response is on-topic for the given utterance; informativeness evaluates systems' ability to provide new information instead of copying from the utterance (Zhou et al., 2018a). All responses of sampled 100 cases are selected from four methods with better performances: CCM, GPT-2 (conv), ConceptFlow, and Golden Response. The responses are scored from 1 to 4 by five judges (the higher the better). Table 3 presents Average Score and Best@1 ratio from human judges. The first is the mean of five judges; the latter calculates the fraction of judges that consider the corresponding response the best among four systems. ConceptFlow outperforms all other models in all scenarios, while only using 30% of parameters compared to GPT-2. This demonstrates the advantage of explicitly modeling conversation flow with structured semantics. The agreement of human evaluation is tested to demonstrate the authenticity of evaluation results. We first sample 100 cases randomly for our human evaluation. Then the responses from four better conversation systems, CCM, GPT-2 (conv), ConceptFlow and Golden Responses, are provided with a random order. A group of annotators are asked to score each response ranged from 1 to 4 according to the quality on two testing scenarios, appropriateness and informativeness. All annotators have no clues about the source of generated responses. The agreement of human evaluation for CCM, GPT-2 (conv) and ConceptFlow are presented in Table 4. For each case, the response from ConceptFlow is compared to the responses from two baseline models, CCM and GPT-2 (conv). The comparison result is divided into three categories: win, tie and loss. Then the human evaluation agreement is calculated with Fleiss' Kappa (  ). The  value ranges from 0.21 to 0.40 indicating fair agreement, which confirms the quality of human evaluation. Both automatic and human evaluations illustrate the effectiveness of ConceptFlow. The next experiment further studies the effectiveness of multi-hop concepts in ConceptFlow. This part explores the role of multi-hop concepts in ConceptFlow. As shown in Figure 3, three experiments are conducted to evaluate the performances of concept selection and the quality of generated responses with different sets of concepts. This experiment considers four variations of outer concept selections. Base ignores two-hop concepts and only considers the central concepts. Rand , Distract , and Full add two-hop concepts in three different ways: Rand selects concepts randomly, Distract selects all concepts that appear in the golden response with random negatives (dis-tractors), and Full is our ConceptFlow (select) that selects concepts by learned graph attentions. As shown in Figure 3(a), Full covers more golden concepts than Base . This aligns with our motivation that natural conversations do flow from central concepts to multi-hop ones. Compared to Distract setting where all ground truth two-hop concepts are added, ConceptFlow (select) has slightly less coverage but significantly reduces the number of two-hop concepts. The second experiment studies the model's ability to generate ground truth concepts, by comparing the concepts in generated responses with those in ground truth responses. As shown in Figure 3(b), though Full filtered out some golden two-Depth Amount Golden Coverage Ratio Number Zero-hop 5.8 9.81% 0.579 + One-hop 98.6 38.78% 2.292 + Two-hop 880.8 61.37% 3.627 + Three-hop 3769.1 81.58% 4.821 ConceptFlow 198.6 52.10% 3.075 Table 5: Statistics of Concept Graphs with different hops, including the total Amount of connected concepts, the Ratio and Number of covered golden concepts (those appear in ground truth responses). hop concepts, it outperforms other variations by large margins. This shows ConceptFlow's graph attention mechanisms effectively leverage the pruned concept graph and generate high-quality concepts when decoding. The high-quality latent concept flow leads to better modeling of conversations, as shown in Figure 3(c). Full outperforms Distract in their generated responses' token level perplexity, even though Distract includes all ground truth two-hop concepts. This shows that negatives selected by ConceptFlow, while not directly appear in the target response, are also on-topic and include meaningful information, as they are selected by graph attentions instead of random. More studies of multi-hop concept selection strategies can be found in Appendix A.2. As shown in Table 5, the Number of covered golden concepts increases with more hops. Compared to zero-hop concepts, multi-hop concepts cover more golden concepts, confirming that conversations naturally shift to multi-hop concepts: extending the concept graph from one-hop to two-hop improves the recall from 39% to 61%, and to three-hop further improves to 81%. However, at the same time, the amounts of the concepts also increase dramatically with multiple hops. Three hops lead to 3,769 concepts on average, which are 10% of the entire graph we used. In this work, we choose two-hop, as a good balance of coverage and efficiency, and used ConceptFlow (select) to filter around 200 concepts to construct the pruned graph. How to efficiently and effectively leverage more distant concepts in the graph is reserved for future work. Some cases from three conversation models are listed in Table 6. Responses from CCM may repeat the same contents as it does not explicitly model the traverse in the concept space. For example, the responses from the first and third cases always repeat I'm not sure. On the other hand, GPT-2 generates more fluent responses compared to CCM. Nevertheless, some cases from GPT-2 merely copy contents or concepts from the given post. For example, for the third case, GPT-2 (conv) mainly discusses the concept music. In comparison, the generated responses from our ConceptFlow are more fluent and informative than those from both CCM and GPT-2. For example, in the third case, ConceptFlow brings associated concepts sound and check to the response generation, hopping from the grounded concepts mu-sic and advice. Introducing these multi-hop concepts effectively improves the informativeness and diversity of generated responses. Figure 4 presents a case study of ConceptFlow. The attention score  e i and  f are presented in the form of color intensity. The championship of zero-hop, fan of one-hop and team of two-hop receive more attention than others and are used to #1 Post actually i stayed at the building right next to the lighthouse . generate the response. The concept flow from fans to fan models the concept shift from user post to response. The concept flow from fan to team further describes the concept shift in response generation. In addition, some concepts, such as win and pretty, share higher attention and may help to understand the one-hop concepts, and are filtered out when generating response by the gate   according to the relevance with conversation topic. This experiment studies the learned attention of ConceptFlow on different groups of concepts. We consider the average attention score (  for central concepts and  (Appendix A.4) for two-hop concepts) from all decoding steps. The probability density of the attention is plotted in Figure 5. Figure 5(a) shows the attention weights on central concepts. ConceptFlow effectively attends more on golden and zero-hop concepts, which include more useful information. The attention on two-hop concepts are shown in Figure 5(b). ConceptFlow attends slightly more on the Golden two-hop concepts than the rest two-hop ones, though the margin is smallerthe two-hop concepts are already filtered down to high-quality ones in the ConceptFlow (select) step. ConceptFlow models conversation structure explicitly as transitions in the latent concept space, in order to generate more informative and meaningful responses. Our experiments on Reddit conversations illustrate the advantages of ConceptFlow over previous conversational systems. Our studies confirm that ConceptFlow's advantages come from the high coverage latent concept flow, as well as its graph attention mechanism that effectively guides the flow to highly related concepts. Our human evaluation demonstrates that ConceptFlow generates more appropriate and informative responses while using much fewer parameters. In future, we plan to explore how to combine knowledge with pre-trained language models, e.g. GPT-2, and how to effectively and efficiently introduce more concepts in generation models. Houyu Zhang, Zhenghao Liu and Zhiyuan Liu is supported by the National Key Research and Development Program of China (No. 2018YFB1004503) and the National Natural Science Foundation of China (NSFC No. 61772302, 61532010). We thank Hongyan Wang, Shuo Wang, Kaitao Zhang, Si Sun, Huimin Chen, Xuancheng Huang, Zeyun Zhang, Zhenghao Liu and Houyu Zhang for human evaluations.
abstain objective abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective result method other other other other other other other other other other method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method objective objective other other