We consider neural language generation under a novel problem setting: generating the words of a sentence according to the order of their first appearance in its lexicalized PCFG parse tree, in a depth-first, left-to-right manner. Unlike previous tree-based language generation methods, our approach is both (i) top-down and (ii) explicitly generating syntactic structure at the same time. In addition, our method combines neural model with symbolic approach: word choice at each step is constrained by its predicted syntactic function. We applied our model to the task of dialog response generation, and found it significantly improves over sequence-to-sequence baseline, in terms of diversity and relevance. We also investigated the effect of lexicalization on language generation, and found that lexicalization schemes that give priority to content words have certain advantages over those focusing on dependency relations. Neural encoder-decoder architectures have shown promise and become very popular for natural language generation. Over the past few years, there has seen a surging interest in sequence-to-sequence learning for dialog response generation using neural encoder-decoder models (Vinyals and Le, 2015; Serban et al., 2017). Typically, an encoder encodes conversational context (source side) information into vector representations, and a decoder auto-regressively generates word tokens conditioned on the source vectors and previously generated words. Two problems arise with the standard left-to-right decoding mechanism. First, no future information is available at any step of the decoding process, while the study of linguistic dependency structure shows that certain words depend on the others that come right to them. Second, preceding words define the context for following words in left-to-right, auto-regressive language models, while linguistic theories may prefer other hierarchies (e.g., adjectives modifying nouns, adverbs modifying verbs). Psycho-linguistics studies also suggest that human may first generate the abstract representation of the things to say, and then linearize them into sentences (Dell et al., 1999). Therefore, it is appealing to consider language generation in alternative orders. This poses a greater challenge because a mechanism in extra to word generation is needed for deciding the position of each word. Some recent works adopt a syntax-free approach to address this problem. (Mehri and Sigal, 2018) proposed a middle-out decoder that starts from the middle of sentences and finishes the rest in forward and backward directions. (Mou et al., 2016) and (Li and Sun, 2018) start with one or two predicted keywords and generate the rest of sentences in a similar fashion. Others incorporate tree structures without syntactic relations and categories. (Zhou et al., 2018) canonicalizes the dependency structures of sentences into ternary trees, and generate only the words top-down. Yet another line of work aim to model the full syntactic trees. (Gu et al., 2018) generates phrase structures and part-of-speech tags along with words for machine translation. (Dyer et al., 2016) generates shift-reduce action sequences of context-free grammars in addition to words for language model and parsing. But words are still generated in left-to-right order in their approaches. In the domain of dialog, we believe language generation can benefit from alternative orders, for the same reasons argued earlier. On the other hand, in human conversations, the structure of utterances usually correspond with dialog states (e.g., wh-noun or wh-adverb phrases are more likely to be used in a request state), so modelling phrase structures can potentially help capturing discourse level information. In order to be able to generate complete syntactic trees, while be flexi-ble about word generation order at the same time, the use of lexicalized grammar becomes a natural choice. Recent years has seen works in language model and generation through alternative orders. (Zhang et al., 2016) developed a top-down neural architecture for language model that alternates between four LSTM decoders according to given dependency relations. (Ford et al., 2018) proposed a two-stage language model, of which the first stage is a language model that generates templates, and the second stage is a translation model that fills in the blanks. Word generation order varies with the choice of words that are generated at different stages. Language generation with tree structures has been explored more thoroughly for neural machine translation. (Eriguchi et al., 2017) and (Aha-roni and Goldberg, 2017) generate CFG trees in bracketed form. (Wu et al., 2017) generates the sequence of transitions to form dependency trees. More recent works have focused on explicitly generating tree structures (Wang et al., 2018; Gu et al., 2018). Regarding neural architectures for tree generation in the field of natural language processing, (Dong and Lapata, 2016) and (Yin and Neu-big, 2017) use a single decoder with parent-feeding mechanism to generate logical forms and programming codes. (Gu et al., 2018) applied the doubly-recurrent neural networks of (Alvarez-Melis and Jaakkola, 2016) with attention mechanism to machine translation. Their model uses two decoders, of which one memorizes the ancestors, and the other remembers the siblings. (Wang et al., 2018) also uses two decoders, but one for generating words and the other for generating syntax trees. In the domain of dialog response generation, the use of syntactic structures is under-studied. (Mou et al., 2016) and (Li and Sun, 2018) considered starting with keywords, and finish the sentences in forward and backward directions. Their models in principle are not tree-structured. The closest thing to our knowledge is by (Zhou et al., 2018). They proposed to convert dependency trees to ternary trees, but ignore the type of dependency relations. In other words, they modelled on trees of which the nodes and edges have no labels. The key difference between their approach and ours is that we generate syntax trees with labels, and word choices are also constrained by the labels. We first consider the following three requirements when generating an L-PCFG syntax tree: Deciding the structure of children. Several mechanisms have been proposed for deciding the structure of children of each node in the context of tree generation. One of them decide tree topology by using a sequence model to generate children one by one and predict stopping tokens (Alvarez-Melis and Jaakkola, 2016). Then there is a simpler approach that treats each combination of the labels of the children as one token, and predict such tokens when generating the parent node (Yin and Neubig, 2017). For language generation, we adopt the second approach and predict the combination of the labels of children, i.e. the rules, for two reasons: (i) the space of grammar rules is generally sparse even when its dimensionality is exponential of the number of labels, and (ii) with sequential generation of labels, as in the first approach, it is hard to enforce the labels of the children to form a valid grammar rule. Deciding the heir of a node. Recall the definition of lexicalized PCFG: let W , N , R be the sets of lexicons, labels, and rules, where each rule is one of the following forms:  X ( h )  h  X ( h )  Y 1 ( h 1 ) . . . Y k ( h k ) such that there exists i , h i = h . where X, Y 1 , . . . , Y k  N , h, h 1 , . . . , h k  W . We do not restrict ourselves to Chomsky Normal Form, and rules can have any number of children. The i th children in the second case is called the heir . The key difficulty to top-down generation of lexicalized PCFG parse tree is deciding which child would be the heir. One way is to make explicit decision to select the child by adding a switch variable, at the cost of increasing the complexity of the problem. Instead, we make a change to the second case above, and simplify the problem by restricting the rules to be of the following form: Figure 1: The first tree is the result of Stanford parser. In other words, the heir would inherit both the lexicalization and the label of the parent (with the possible exception that the root node may produce children that are not labeled with root). When generating a parent node and its children, we restrict the choice of rules to those containing the label of the parent, so the heir can be inferred from the chosen rules by looking for the child that has the same label as its parent (in case there are multiple children that have the same label, we choose the rightmost one; other heuristics are possible). Note that under such restriction we end up with parse trees in which all labels are part-of-speech tags. Sequentialization of a tree. Previous works adopt various construction orders of trees. (Zhang et al., 2016), (Zhou et al., 2018), (Alvarez-Melis and Jaakkola, 2016), and (Gu et al., 2018) generate trees through level-order traversal (breadth-first, left-to-right), whereas (Yin and Neubig, 2017) and (Wang et al., 2018) generate trees through preorder traversal (depth-first, left-to-right). (Kun-coro et al., 2018) also experimented with bottom-up and left-corner construction orders for language model. While finding the optimal order of generating trees is beyond the scope of this work, we follow (Yin and Neubig, 2017) and (Wang et al., 2018), and generate lexicalized PCFG syntax trees through pre-order traversal. In this paper, we give the following graph-theoretic definition of L-PCFG syntax trees. Let W and N be the sets of lexicons and labels. Let R = (cid:83)  k =1 N k be the set of production rules. Then an L-PCFG syntax tree T is an ordered tree defined by the triple of vertices v  V  W  N  R  N , edges e  E  V  V , and bijection f : V  N +  V such that ( v, f ( v, j ))  E , where j range from 1 to the number of children of v . The fourth coordinate of v is the index of its heir, that is, ( w, n, r, i ) = f (( w 0 , n 0 , r 0 , i 0 ) , i 0 ) =  w = w 0 , n 0 = n We say a node v = ( w, n, r, i ) is a leaf if r is unary: v is a leaf  r  N The parent of v k is denoted by v p ( k ) = ( w p ( k ) , m p ( k ) , n p ( k ) , i p ( k ) ) . Following previous work, we sequentialize L-PCFG parse trees and generate its content in an auto-regressive manner. When generating k th node, we predict the lexicalization w k and the rule r k . The pre-order history available when generating k th node is n 1    n k  1 , w 1    w k  1 , r 1    r k  1 , denoted by H k . The label ROOT is given at the start of generation. The label of the k th node is inferred from the production rule of its parent and the order of k th node among its siblings, and is used as input together with H k . When a leaf node is reached, the program backtraces until it finds an ancestor that has unfinished child, and proceeds to the first such child. We factor the joint probability of w k and r k into two component: a word model and a syntax model, as follow: P ( w k , r k | n k , H k ) = P ( w k | n k , H k )  P ( r k | n k , H k ) The details of both models are given in the following sections. We parse the responses in training corpus using the lexicalized parser by (Klein and Manning, 2003) (which we call Stanford parser for the rest of this paper). We then replace the label of each node with that of their heir in a bottom-up manner. Unary rules at non-leaf nodes are removed as they become redundant given our definition of lexiclaized PCFG. Stanford parser lexicalizes PCFG phrase structures by looking for the most likely combination of phrase structure and dependency structure. While their approach is optimized for parsing, the syntax trees lexicalized this way has a drawback for the purpose of generation. Empirically, their parser tends to lexicalize the first few nodes with auxiliary verbs or common verbs (e.g. be, must), and in some cases prefer function words over content words (e.g. in preposition phrases). We hypothesize that choosing content words over functions, or infrequent words over frequent words as lexicalization heads will help making the generation more specific and meaningful. Hence, we consider two alternative lexicalization schemes: Content-based lexicalization. We rank words according to their part-of-speech in the sentence by the following order: nouns > verbs = adjectives > adverbs > everything else. If two words have the same rank, we give priority to the rightmost one. See Figure 1 for an example. Frequency-based lexicalization. We ignore part-of-speech information and rank all words by their frequencies. We regard less frequent words as more important. To represent the state of a tree node by encoding its pre-order history H k , we use 3 LSTMs to memorize memorize the lexical and grammatical contents in H k . Encoding lexicalization history. We use 2 LSTMs to encode the lexicalization history, i.e. w 1    w k  1 : a surface decoder, L s , which takes the lexicalization of the leaves in the history as inputs; and an ancestor decoder, L a , which is given the lexicalization of the ancestors of the current node. This is another form of doubly-recurrent neural networks. Different from (Alvarez-Melis and Jaakkola, 2016), we chose to encode leaves instead of siblings. Denote the lexicalizaiton of leaves and ancestors in H k by { w l ( k ) i } and { w a ( k ) i } . We show that for an L-PCFG syntax tree, { w l ( k ) i } and { w a ( k ) i } sufficiently cover the lexical content of H k : Proposition. For any index set I k  { 1 , 2 , . . . , k  1 } , if w n  { w j } j  I k for all n < k , then { l ( k ) i } (cid:83) { a ( k ) i }  I k , i.e. { l ( k ) i } and { a ( k ) i } together is the minimal index set to cover w 1    w k  1 . Encoding syntactic history. We encode the previous labels and rules using the full history with a single LSTM, L g . At step k , the input to L g is the concatenation of the embeddings of n k , the rule of parent node r p ( k ) , and depth of the node d . The depths of nodes deeper than 10 are rounded down to 10. We adopt attention mechanism into our architecture for response generation. We use a one-layered LSTM to encode the dialog history, which is the concatenation of the last few utterances. The initial hidden states of L s , L a , and L g is computed from the last hidden state of source encoder using 2 fully-connected layers with rectified linear activation. At time step t , the concatenation of the hidden states of L s and L a at step t  1 is used as the key for querying the source. The attention weights are the inner products of the key and the hidden states at source side, normalized by softmax function. The weighted sum of source hidden states results in the attention context, c ( k ) 4.6 Decoding Denote the hidden states of L s , L a , L g at node k as h s ( k ) , h a ( k ) , h g ( k ) . Denote the softmax func-Figure 2: Demonstration of inputs and outputs at node DT. The sequence of inputs to each encoder are shown in the graph. The inputs to L g is a sequence of labels, rules of parents, and tree depths (only labels are shown). L s and L a are used for predicting the word for DT. L s and L g are used for predicting the rule for DT. RULE: DT indicates DT will be a leaf node since the number of symbols is 1. In this tree, words are generated in the order: daughter I have a. tion by  . E w  R | W | d w and E r  R | P | d r are embedding matrices for words, labels, and rules. A w  R n w  d w and A r  R n p  d r are weight matrices ( n w , n p are the dimensions of input neu-rons). We use weight tying (Press and Wolf, 2017) to limit the search space for parameters. Word prediction. To decode for w k , we use the the hidden states of surface decoder L s and ancestor decoder L a . If v k is a heir, then P ( w k | n k , H k ) = (cid:40) 1 w k = w p ( k ) 0 w k (cid:54) = w p ( k ) Otherwise, the probability of w k is given by: P ( w k | n k , H k ) =  ( tanh ([ h s ( k ); h a ( k ); c ( k )] A w ) E Tw ) At decoding time, we impose an additional constraint that w k be a valid word for label n k , to enforce grammaticality. This is estimated from the co-occurrence of w k and n k in the tagged training corpus. We only use those words whose frequency of co-occurrence with the given label is above a certain threshold. Given the definition of L-PCFG syntax tree, we only consider rules that contain n k at decoding time. There is one exception: at ROOT node, only unary rules are considered, and they do not have to contain the label ROOT. Hence, we train our architecture by minimizing the negative log-likelihood of words and rules:  log P ( T ) =  1 | W ( T ) | (cid:88) k v k (cid:54) = f ( v p ( k ) ,i p ( k ) ) log P ( w k | n k , H k )  1 | T | (cid:88) k log P ( r k | n k , H k ) where | W ( T ) | is the number of non-heir nodes (or the number of words in the original sentence), and | T | is the number of nodes in T . Note that the log probability of each word in the sentence appears exactly once in the above equation. At test time, we conduct beam search and use the same equation to score each generation for selecting words and rules. In our experiments, we use unlexicalized PCFG as an additional baseline. We still replace the labels of each node with their heirs' in the parse tree returned from Stanford parser, but words are generated only at leaf nodes. This baseline has syntactic structure while generating words from left to right. We use it as a test against top-down generation of words with syntax. All models are implemented using PyTorch. The hidden size of all LSTM encoders and decoders are 512. The size of embeddings of words, labels, rules, and tree depth are 300. We trained our models using stochastic gradient descent (SGD) with momentum and exponential learning rate decay. Dropout is applied to the input and output layer of LSTMs. We evaluate our model for dialog response generation on Persona dataset ((Zhang et al., 2018)). Each person is give a list of persona descriptions in simple sentences, and they are required to con-verse according to the given persona. We use last 3 utterances for each response as source. We prepend persona descriptions to source. We use global attention over persona descriptions to compute context vectors. During pre-processing, we truncate all trailing punctuations. We measure how early do each type of words appear in different generation orders  standard left-to-right order, dependency-based lexicalization (as in Stanford parser), and content-based lexicalization. The earlier a word appear, the less context there is for predicting it. As shown in Table 1, content-based lexicalization can make nouns and verbs appear much earlier, while delaying function words. To verify frequency-based lexicalization is making infrequent words appear earlier, we show the average frequencies of the first five words. The first few words are more important since they decide the context for generating the following words. In Table 2, the first two words of each parse tree under frequency-based lexicaliza-Seq2seq Ours Standard 3.682 N/A Dependency 4.015 3.964 Content 4.115 3.865 Frequency 4.088 3.827 Table 3: Perplexities. 5.3 Perplexity We compare per word likelihood given different generation orders and architectures. For left-to-right sequence decoder, the non-standard generation orders are obtained by linearizing L-PCFG parse trees in pre-order traversal, and words of heirs are not repeated in the linearization. Note that our word model generates word without using rules and labels as inputs to its networks. As can be seen from Table 3, alternative word generation orders all make it harder for standard left-to-right sequence decoder to learn to predict the next word. On the other hand, using a doubly-recurrent architecture, specifically the surface decoder and the ancestor decoder, can improve perplexity scores for top-down word generation over the left-to-right decoder. While our word model with top-down word generation orders has higher perplexity scores than simple model with standard generation order, we emphasize that perplexity is not an appropriate measure for generation tasks. Word Overlap Based Metrics. We use BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and METEOR (Banerjee and Lavie, 2005) scores as automatic evaluation metrics. While the reliability of such metrics has been criticized (Liu et al., 2016), there is also evidence that for task-oriented domains, these metrics correlate with human judg-ment to a certain extent (Sharma et al., 2017). Word Embedding Based Metrics. We evaluate the semantic similarity between generated responses and human responses/persona by the cosine distance of their sentence embeddings. We use the word averaging approach by (Arora et al., 2016) to embed the responses, which has been demonstrated to be very good at capturing lexical level semantic similarity. The normalizing singular vector is obtained from the responses in training set. Novelty and diversity. We measure word overlapping between generated responses and the responses in training set using BLEU and ROUGE as a proxy for novelty. The responses in training set with most common words with generated responses are used as references. For diversity, we count the number of distinct n-grams. In addition, we perform a k-means clustering on the sentence embeddings of responses into 10 clusters, and measure average squared Euclidean distance between members of each cluster (Inertia). The larger the number, the harder it is to separate embeddings into 10 clusters, thus the greater the diversity. Evaluation results are shown in Table 4. For metrics that are not measured using ground truth response as reference, we consider the closer to the number for human responses the better. We first look at measures for overall generation quality. We can see modelling syntactic structures is capable of generating longer responses. BLEU scores are positively correlated with lengths. While syntactic models do better on BLEU, and slightly better on METEOR than sequence-to-sequence baseline, they are generally not on par with the baseline in terms of ROUGE-L, except for frequency-based lexicalization. Among grammar models, lexicalized grammars out-performed unlexicalized grammar. Relevance is measured using cosine similarities with the previous utterance and persona. Syntactic models with lexicalized grammar beat the baseline in terms of relevance. Furthermore, content-based lexicalization is much more on topic with the last source utterance than dependency-based L g + L s L g + L s + L a L g Lengths 11.04 7.95 8.46 Distinct uni-gram 813 383 376 Distinct bi-gram 5427 1763 1775 Distinct tri-gram 10787 3061 3522 Inertia 3756 2049 2346 BLEU on training 0.5402 0.8056 0.6612 ROUGE on training 0.6226 0.8543 0.7417 Table 6: Ablation studies. and frequence-based lexicalization. Dependency-based lexicalization is best at being adherent to personas than the other two lexicalization schemes. All syntactic models generate more novel responses than sequence-to-sequence baseline, as reflected in the last two rows in 4. This is consistent with the observation that sequence-to-sequence model exhibits retrieval-like behaviour, selecting what is most common in the training corpus. Syntactic models also have larger vocabularies. As for cluster analysis, unlexicalized grammar model and dependency-based lexicalized grammar model have better diversity than sequence-to-sequence model; content-based and frequency-based lexicalization have slightly smaller inertia than the baseline. We present a few examples generated by sequence-to-sequence baseline and L-PCFG model. There is a clear difference of how left-to-right decoder and L-PCFG tree decoder do conjunctions. Most of the time, standard LSTM decoder combine sentences with periods, while tree decoders learn to use conjunction words, or even clauses. We also performed an error analysis on the generated responses by L-PCFG, and in Table 7 we selected the most peculiar and representative. These examples are all syntactically fine, but they do not follow the convention of the language or common sense. The first example contains the most common errors in the responses generated by L-PCFG: misuse of prepositions and determiners. It can be fixed by replacing as a with for. The error of other two examples have even less to do with syntax. The second one misuses the verb be, which is probably caused by the high frequency of the word in the corpus. The error of the third example is beyond surface level. Note that phrases such as cooking as a dinner and be a dog never appear in the corpus. It is clear that L-PCFG models are learning to make compositions of words and phrases, unlike standard LSTM decoder, which seems to only memorize word combinations. i am doing well . just finished cooking as a dinner i am sure it is nice . i am going to be a dog i like to ride my horses on my bike Table 7: Some peculiar examples generated by L-PCFG models. We perform ablation studies on our architecture, with content-based lexicalization. Specifically, we consider two alternative ways of making rule prediciton. The first one takes only the hidden state of L g for predicting rules, hence making the prediction of rules entirely independent from words. The second one takes both the hidden states of L s and L a , together with L g , for predicting rules, in which way future lexical information is used for the construction of syntax trees. For rule prediction using the hidden states of both surface decoder and ancestor decoder, we noticed a significant drop in diversity in generated responses. The model over-predicts what do you do for a living for than 50% of the time, and the lengths of responses tend to be shorter. For the other choice in which rule prediction is independent from words, the results are closer to the original model, but there are still some decreases in lengths and lexical diversity. Upon manual inspection, we found that this model behaved more like sequence-to-sequence model. There are less compound sentences, and more conjunctions of simple sentences by end punctuation marks. also larger. David Alvarez-Melis and Tommi S Jaakkola. 2016. Tree-structured decoding with doubly-recurrent neural networks. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2016. A simple but tough-to-beat baseline for sentence embeddings. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization , pages 6572. Gary S Dell, Franklin Chang, and Zenzi M Grif-fin. Connectionist models of language production: Lexical access and grammatical encoding. Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , volume 1, pages 3343. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A Smith. 2016. Recurrent neural network grammars. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 199209. Akiko Eriguchi, Yoshimasa Tsuruoka, and Kyunghyun Cho. 2017. Learning to parse and translate improves neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , volume 2, pages 7278. Nicolas Ford, Daniel Duckworth, Mohammad Norouzi, and George Dahl. 2018. The importance of generation order in language modeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 29422946. Dan Klein and Christopher D Manning. 2003. Fast exact inference with a factored model for natural language parsing. In Advances in neural information processing systems , pages 310. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out . We consider the problem of generating natural language in alternative orders and with syntactic tree structures, with the use of lexicalized grammar. By incorporating syntactic structures, our models are capable of generating longer sentences. By changing lexicalizaion schemes and making content words appear earlier in generation process, our models are able to make word choices that are more relevant to source. Furthermore, incorporating syntax facilitates response generation with richer vocabularies and more complex structures. On the other hand, as shown in our error analysis, there is still room for improvement on discourse and pragmatics level. This material is based upon work partially supported by the National Science Foundation (Award # 1722822). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of National Science Foundation, and no official endorsement should be inferred.
objective abstain abstain abstain method result objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain other other other other other other other other other other other other other other other abstain other other method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain other abstain abstain method abstain abstain abstain method method method abstain abstain abstain abstain method method method method abstain abstain abstain method method method method abstain abstain method method method abstain abstain method abstain method abstain method other abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain method method abstain method abstain method method method abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain result other other