Syntax has been demonstrated highly effective in neural machine translation (NMT). Previous NMT models integrate syntax by representing 1-best tree outputs from a well-trained parsing system, e.g., the representative Tree-RNN and Tree-Linearization methods, which may suffer from error propagation. In this work, we propose a novel method to integrate source-side syntax implicitly for NMT. The basic idea is to use the intermediate hidden representations of a well-trained end-to-end dependency parser, which are referred to as syntax-aware word representations (SAWRs). Then, we simply concatenate such SAWRs with ordinary word embeddings to enhance basic NMT models. The method can be straightforwardly integrated into the widely-used sequence-to-sequence (Seq2Seq) NMT models. We start with a representative RNN-based Seq2Seq baseline system, and test the effectiveness of our proposed method on two benchmark datasets of the Chinese-English and English-Vietnamese translation tasks, respectively. Experimental results show that the proposed approach is able to bring significant BLEU score improvements on the two datasets compared with the baseline, 1.74 points for Chinese-English translation and 0.80 point for English-Vietnamese translation, respectively. In addition, the approach also outperforms the explicit Tree-RNN and Tree-Linearization methods. In the past few years, neural machine translation (NMT) has drawn increasing interests due to its simplicity and promising performance (Bah-danau et al., 2014; Jean et al., 2015; Luong and Manning, 2015; Luong et al., 2015; Shen et al., 2016; Vaswani et al., 2017). The widely used  Corresponding author. An example of input dependency tree. sequence-to-sequence (Seq2Seq) framework combined with attention mechanism achieves significant improvement over the traditional statistical machine translation (SMT) models on a variety of language pairs, such as Chinese-English (Shi et al., 2016; Mi et al., 2016; Vaswani et al., 2017; Cheng et al., 2018). Under an encoder-decoder architecture, the Seq2Seq framework first encodes the source sentence into a sequence of hidden vectors, and then incrementally predicts the target sentence (Cho et al., 2014a). Recently, inspired by the success of syntax-based SMT (Williams et al., 2016), researchers propose a range of interesting approaches for exploiting syntax information in NMT models, as syntactic trees could offer long-distance relations in sentences (Shi et al., 2016; Wu et al., 2017b; Li et al., 2017; Bastings et al., 2017; Hashimoto and Tsuruoka, 2017). As a straightforward method, tree-structured recurrent neural network ( Tree-RNN ) can elegantly model the source-side syntax and globally encode the whole trees. Eriguchi et al. (2016), Chen et al. (2017a) and Yang et al. (2017) show that Tree-RNN can effectively integrate syntax-oriented trees into Seq2Seq NMT models. Regardless of the effectiveness of Tree-RNN, we find that it suffers from a severe low-efficiency problem because of the heterogeneity of different syntax trees, which leads to increasing difficul-ties for batch computation compared with sequential inputs. Even with deliberate batching method of Neubig et al. (2017), our preliminary experiments show that Tree-RNN with gated recurrent unit (GRU) can lead to nearly four times slower performance when it is integrated into a classical Seq2Seq system. To solve the problem, Tree-Linearization is a good alternative for syntax encoding. The main idea is to linearize syntax trees into sequential symbols, and then exploit the resulting sequences as inputs for NMT. Li et al. (2017) propose a depth-first method to traverse a constituent tree, converting it into a sequence of symbols mixed with sentential words and syntax labels. Similarly, Wu et al. (2017b) combine several strategies of tree traversing for dependency syntax integration. In this work, we present an implicit syntax encoding method for NMT, enhancing NMT models by syntax-aware word representations ( SAWRs ). Figure 1 illustrates the basic idea, where trees are modeled indirectly by sequential vectors extracted from an encoder-decoder dependency parser. On the one hand, the method avoids the structural heterogeneity and thus can be integrated efficiently, and on the other hand, it does not require discrete 1-best tree outputs, alleviating the error propagation problem induced from syntax parsers. Concretely, the vector outputs are extracted from the encoding outputs of the encoder-decoder dependency parser. As shown in Figure 1, the encoding outputs, denoted as o = o 1    o 6 , are then integrated into Seq2Seq NMT models by directly concatenated with the source input word embeddings after a linear projection. We start with a Seq2Seq baseline with attention mechanism (Bahdanau et al., 2014) for study, following previous studies of the same research line, and then integrate source dependency syntax by SAWRs. We conduct experiments on Chinese-English and English-Vietnamese translation tasks, respectively. The results show that our method is very effective in source syntax integration. With source dependency syntax, the performances of Chinese-English and English-Vietnamese translation can be significantly boosted by 1.74 BLEU points and 0.80 BLEU points, respectively. We also compare the method with the representative Tree-RNN and Tree-Linearization approaches of syntax integration, finding that our method is able to achieve larger improvements than the two approaches for both tasks. All the codes are released publicly available at https://github.com/zhangmeishan/SYN4NMT under Apache License 2.0. We take the simple yet effective Seq2Seq model with attention mechanism proposed by Luong et al. (2015) as our baseline. Under the standard encoder-decoder architecture, an encoder first maps the source-language input sentence into a sequence of hidden vectors, and a decoder then incrementally predicts the target output sentence. In particular, we should notice that several recent models (Vaswani et al., 2017; Zheng et al., 2017; Cheng et al., 2018) which have been shown to be more powerful can also serve as our baseline, since these models focus on very different aspects of NMT, which could be potentially complementary with our focus of syntax integration. We will demonstrate it by experimental analysis as well. In the encoder part, a single-layer bi-directional recurrent neural network (Bi-RNN) is employed to encode the sentence in order to capture features from the current word and the unbounded left and right contextual words. Given a source-language input sentence x = x 1    x n and its embedding sequence e x 1    e x n , the Bi-RNN produces an encoding sequence of dense vectors h = h 1    h i    h n : h i =  h i   h i ,  h i = rnn L ( e x i ,  h i  1 )  h i = rnn R ( e x i ,  h i +1 ) (1) where rnn L /R can be either GRU (Cho et al., 2014b) or LSTM. We use GRU all through this paper for efficiency following Chen et al. (2017a). The decoder part incrementally predicts the target word sequence y = y 1    y m , whose translation probability is defined as follows: The training objective is to maximize the probability of the reference translation. During evaluation, we aim to search for a target sentence with the highest probability for a given source sentence. The probability of the j -th target word is computed by a two-layer feed-forward neural network: p ( y j | y 1    y j  1 , h ) = g ( s j  1 , c j ) , (3) where s j  1 = rnn tgt ( e y j  1  c j  1 , s j  2 ) is the output of a left-to-right RNN over the predicted words, and the c j / c j  1 is the weighted sum over the encoding sequence h of the source sentence via the attention mechanism, which is computed as follows: c j = n (cid:88) k =1  j,k h k  j,k = exp(  j,k ) (cid:80) nl =1 exp(  j,l )  j,l = s T j  1 W a h l (4) where W a is the model parameter in attention. Syntax information has been demonstrated to be valuable for NMT. Previously, there were two representative approaches to encode syntax into an NMT model. The first approach directly represents an input syntax tree by Tree-RNN , and then uses the Tree-RNN outputs as additional encoder inputs for NMT. The second approach models source syntax trees indirectly by first converting a hierarchical tree into a sequence of symbols, and then use the symbols as inputs for NMT. The second method is referred to as Tree-Linearization here. Tree-RNN is able to represent the syntax structures fully and comprehensively. However, because of the heterogeneity of different syntax trees, this approach suffers serious inefficiency encoder decoder input embedding projection Bi-RNN LLL decoder parser output translation output encoder Figure 2: The framework of the SAWR approach, where the left part shows the encoder-decoder of a supervised dependency parsing model and the right part shows the NMT encoder-decoder. problem as the increased difficulty of batch computation for GPU neural computation. The second approach exploits an alternative sequence to substitute the original trees, which solves the inefficiency problem. But it may bring loss of syntax information because the hierarchical tree structure is no longer maintained in the new representation, which could be potentially useful for NMT. Both the two syntax integration approaches are based on discrete 1-best outputs of a supervised dependency parser, which may suffer from the error propagation problem. Incorrect syntax trees as inputs for NMT may produce erroneous outputs, leading to inappropriate translation results. In order to alleviate the problem, we present a novel method not using the discrete parsing outputs. We focus on supervised dependency parsing models which can be formalized as an encoder-decoder architecture, and exploit the encoder outputs as the inputs for our Seq2Seq NMT model. The encoder outputs are sequences of dense vectors aligning with the source sentential words, as shown in Figure 1, and thus they could be easily combined with the encoder part of our NMT model. We refer to this method as SAWR for brief. Our approach takes the implicit hidden outputs from a supervised parser as inputs for NMT, which greatly reduces the direct influence brought from discrete 1-best parser outputs. Figure 2 shows the framework of SAWR. Concretely, we first project the encoder outputs of a dependency parsing model into a sequence of vectors by a feed-forward linear layer, as shown by the projection module in Figure 2: s i = W o i + b (5) where o = o 1    o n is the encoder output of a parsing model, W and b are model parameters. Then we concatenate the resulting vectors with the source embeddings as inputs for the baseline Bi-RNN Encoder. Thus the encoder process can be formalized as follows: h = Bi-RNN (cid:0) e x 1  s 1 ,    , e x n  s n (cid:1) . We can train both dependency parsing and machine translation model parameters concurrently. In this work, we focus on the machine translation task and do not involve the training objective of dependency parsing. However, we can still fine-tune model parameters of the encoder part of dependency parsing by back-propagating the training losses of NMT into this part as well. Actually, SAWRs are also similar to the ELMO embeddings (Peters et al., 2018). ELMO learns context word representations by using language model as objective, while SAWRs learn syntax-aware word representations by using dependency parsing as objective. On the other hand, compared with the Tree-RNN and Tree-Linearization methods which encode syntax trees by neural networks directly, SAWRs are less sensitive to the output syntax trees. Thus the SAWR method can alleviate the error propagation problem. Data. We conduct experiments on the Chinese-English and English-Vietnamese translation tasks, respectively. For Chinese-English, we use the parallel training data from the publicly available LDC corpora, 1 with 28.3M Chinese words and 34.5M English words, respectively, consisting of 1.25M sentence pairs, and test model performances on the NIST datasets, using NIST MT02 as the development data, and MT03-06 as test datasets. For English-Vietnamese, we use the standard IWSLT 2015 dataset, 2 which consists of about 133K sentence pairs, and evaluate our models by exploiting 1 LDC2002E18, LDC2003E07, LDC2003E14, Hansards portion of LDC2004T07, LDC2004T08 and LDC2005T06. the TED tst2012 and tst2013 as the development and test datasets, respectively. For the source side sentences, we construct vocabularies of the most frequent 50K words, while for the target side sentences, we apply byte-pair encodings (BPE) (Sennrich et al., 2016) with 32K merges to obtain subword units, and construct the target vocabularies by the most frequent 32K sub-words. During training, we use only the sentence pairs whose source and target lengths both are no longer than 50 and 150 for Chinese-English and English-Vietnamese translations, respectively. Evaluation. We use the case insensitive 4-gram BLEU score as the main evaluation metrics (Papineni et al., 2002), and adopt the script multi-bleu.perl in the Mose toolkit. 3 Significance tests are conducted based on the best-BLEU results for each approach by using bootstrap resampling (Koehn, 2004). Alternatively, in order to compare the effectiveness of our model with other syntax integration methods, we implement a Tree-RNN approach and a Tree-Linearization approach, respectively:  Tree-RNN: We build a one-layer bidirectional Tree-RNN with GRU over input word embeddings, producing syntax-enhanced word representations, which are then fed into the encoder of NMT as basic inputs. The method is similar to the model proposed by Chen et al. (2017a). Tree-Linearization: We first convert dependency trees into constituent trees (Sun and Wan, 2013), and then feed it into the NMT model proposed by Li et al. (2017). Hyperparameters. We set the dimension sizes of all hidden neural layers to 1024, except the input layers for RNNs (i.e. input word embeddings and the projection layer of SAWR), which are set to 512. We initialize all model parameters by random uniform distribution between [  0 . 1 , 0 . 1] . We apply dropout on the output layer of word translation with a ratio of 0.5. We adopt the Adam algorithm (Kingma and Ba, 2014) for parameter optimization, with the initial learning rate of 5  10  4 , the gradient clipping threshold of 5, and the mini-batch size of 80. During translation, we employ beam search for decoding with the beam size of 5. Source-Side Parsing. We employ the state-of-the-art BiAffine dependency parser recently proposed by Dozat and Manning (2016) to obtain the source-side dependency syntax information. The BiAffine parser can also be understood as an encoder-decoder model, where the encoder part is a three-layer bi-directional LSTM over the input words, and the decoder uses BiAffine operations to score all candidate dependency arcs and finds the highest-scoring trees via dynamic programming. For Chinese-English translation, we train the dependency parser on Chinese Treebank 7.0 with Stanford dependencies, 4 using 50K random sentences as the training data and the remaining as the test data. The parser achieves 81 . 02% parsing accuracy (labeled attached score, LAS) on the test dataset. For English-Vietnamese translation, we train the dependency parser on English WSJ corpus, following the same data split as Dozat and Manning (2016), and obtaining a LAS of 93 . 84% on the test dataset. 5 4.2 Speed Comparison All our experiments are run on a single GPU NVIDIA TITAN Xp. We report the averaged one-epoch training time on the Chinese-English translation dataset (consuming all 125M sentence pairs) as follows: Baseline 105 min SAWR 142 min Tree-RNN 498 min Tree-Linearization 137 min 4 https://nlp.stanford.edu/software/stanford-dependencies.shtml 5 For simplicity, we use only words as inputs for both Chinese and English dependency parsing, avoiding the influences brought by other inputs, such as automatic POS tags. The SAWR system spends averaged 142 minutes, 6 37 minutes slower than the baseline model. The Tree-Linearization spends averaged 137 minutes per epoch, which is the fastest syntax integration method. Our SAWR approach spends 5 more minutes than Tree-Linearization, appropriate 3.5% of the total spend time per epoch, which could be negligible. The Tree-RNN model spends 498 minutes per epoch, nearly four times slower than the baseline model. 7 According to the results, we can conclude that the Tree-RNN model is highly inef-ficient for encoding dependency syntax, whereas the SAWR and Tree-Linearization are almost as efficient as the baseline Seq2Seq system. Table 1 shows the main results of all approaches on Chinese-English datasets. Considering the effect of random initialization, we train three individual models for each approach, and use the averaged BLEU scores for fair comparisons. According to the results, we can see that all syntax-integrated approaches can bring significant improvements over the baseline system, which denotes that syntax is highly effective for Chinese-English machine translation. In addition, the proposed SAWR approach obtains the largest BLEU improvements, averaged  = 1 . 74 BLEU points better than the baseline system. The Tree-RNN and Tree-Linearization approaches bring improve-6 We exclude the time consumed by the encoder part of the dependency parsing model for fair comparisons, as other methods require to perform parsing in an offline way. 7 The Tree-RNN model is implemented with deliberate batching motivated by Neubig et al. (2017), without which the model is intolerably slow, reaching about 1,900 minutes per epoch. ments of averaged  = 1 . 32 and  = 1 . 23 BLEU points, respectively. The results show that our implicit syntax-aware encoding method is better than Tree-RNN and Tree-Linearization. We compare our NMT models with other state-of-the-art methods as well. The results are just for reference since experimental details could be very different. In particular, we list the relative improvements over the corresponding baseline models by integrating syntax structures, which are calculated according to their papers. All these studies exploit lower baselines compared with our models. The Tree-RNN and Tree-Linearization are essentially similar to Chen et al. (2017a) and Li et al. (2017), respectively. As shown, our approaches can still obtain large improvements based on a stronger baseline. Table 2 shows the final results on the IWSLT 2015 English-Vietnamese translation task. The overall tendency is similar to that of Chinese-English translation. The syntax information can boost the translation performances by using any of the three approaches. The SAWR approach gives the best translation performance, significantly outperform the baseline system by  = 0 . 80 BLEU points. While although the other two approaches bring better performances, the improvements are not significant. The results demonstrate the advantage of the proposed implicit SAWR approach. By not using the 1-best parser outputs, our approach can reduce the error propagation problem, thus bring larger improvements with syntax. In particular, we find that the increases of BLEU scores are smaller than that of Chinese-English translation by integrating syntactic features. The averaged BLEU increases are 0 . 55 for English-Vietnamese and 1 . 43 for Chinese-English. The possible reason may be due to that the source English sentences are more grammatically rigorous Parser MT03 MT04 MT05 MT06 Average no Tune 38.42 40.60 38.27 38.04 38.83 Tune 37.33 39.45 36.93 37.03 37.69 Table 3: The influence of fine-tuning parser parameters in the SAWR system. than Chinese sentences. For example, the English functional words such as of and s which indicate the possessive relationship, should be always kept in sentences by standard, while their Chinese correspondence    may be omitted in sentences. In this section, we conduct analysis on Chinese-English translation from different aspects to better understand the SAWR approach of integrating source-side dependency syntax for NMT. The SAWR approach directly uses the encoder outputs of a dependency parser as extra inputs for NMT. In the above experiments, we keep the parser model parameters fixed, letting them unin-fluenced from NMT optimization. Actually, this part can be further fine tuned along with the NMT learning, by treating them as one kind model parameters. Thus there arises a question that whether fine-tuning the parser model parameters can bring better performance. As an interesting attempt, we can simultaneously fine tune the parameters of both the parser and the Seq2Seq NMT model during training. Figure 3 shows the results. We can see that fine-tuning decreases the average BLEU score by 38 . 83  37 . 69 = 1 . 14 significantly. This may be because that fine-tuning disorders the representation ability of the parser and makes its function more overlapping with other network components. This further demonstrates that pretrained syntax-aware word representations are helpful for NMT. Alignment quality is an important metric to illustrate and evaluate machine translation outputs. Here we study how syntax features influence the alignment results for NMT. We approximate the alignment scores by the attention probabilities as shown in Equation 4. 8 For better understanding 8 We aim to offer an intuitive interpretation by a carefully-selected example. In fact, the alignment computation method here may be problematic (Koehn and Knowles, 2017). the effectiveness of syntax, we choose the target-side English word of for comparison, which is a grammatical functional word. Figure 3 shows the alignment probability distributions returned by different approaches. Intuitively, this word should be aligned with the Chinese word   (de). But according to the results, we can see that only the SAWR model distributes a high attention score to it, which is consistent with our intuition. The other three models are all aligned to the source word    (modern) with high confidence over 85%. The possible reason for of being aligned to    (modern) could be due to that of modern is a high-frequency collocation in the training corpora. Here we perform model ensembles to examine the divergences of the three syntax-integration approaches (Zhou et al., 2017b; Denkowski and Neubig, 2017). Intuitively, the hetero-approach ensemble which combines three NMT models of different methods should obtain better performances than homo-approach ensembles which combine three NMT models of the same method, since NMT models of different syntax-integrations approaches have larger divergences. Table 4 shows the results. First, we can see that ensemble is one effective technique to improve the translation performances. More impor-(0 , 10] (11 , 20] (21 , 30] (31 , 40] (41 , 50] > 50 30 34 38 42 BLEU Baseline SAWR Tree-RNN Tree-Linearization Figure 4: The effect of source input length. tantly, the results show that the heterogeneous ensemble achieves averaged BLEU improvements by 43 . 10  41 . 24 = 1 . 86 points, better than the gains achieved by all three homo-approach ensembles, denoting that the three approaches could be mutually complementary in representing dependency syntax, and the resulting models of the three approaches are highly diverse. Intuitively, by introducing the source syntax into the NMT model, relations between long-distance words are explicitly modeled by dependency trees, thus we can expect that models enhanced by source syntax are able to bring better translations for longer sentences. Figure 4 shows the performances of the baseline and all syntax-enriched models in terms of source sentence lengths, where we bin all the MT03-MT06 sentences by their lengths into six intervals. The results show that the BLEU scores are improved significantly when source sentential lengths are over 10, which con-firms our intuition. Finally, we examine how the performance of the dependency parser influences the final translation quality. While the full dependency parser is System MT03 MT04 MT05 MT06 Average/  Transformer 40.45 42.76 40.09 39.67 40.74 SAWR 41.63 43.60 41.68 40.21 41.78/+1.04 Tree-RNN 41.24 43.38 41.04 40.02 41.42/+0.68 Tree-Linearization 41.12 43.02 41.04 39.86 41.26/+0.52 Table 5: Final results based on the transformer. trained on 50K sentences, we retrain three weaker dependency parsers on 30K, 10K and 5K sentences, respectively. Figure 5 shows the NMT BLEU scores and the parsing accuracies. It is clear that the parsing accuracy directly influences the translation quality, indicating the effectiveness and importance of exploiting syntactic information. Here we conduct experiments based on the transformer NMT model (Vaswani et al., 2017), which is a stronger baseline, to further verify the effectiveness of our proposed method. This also demonstrates that the proposed SAWR method does not limit to a certain NMT baseline. Concretely, we extend the bottom word representations by incorporating syntactic encodings s = s 1    s n (shown in Equation 5) into them, and then feed them into the transformer encoder by a linear projection layer to align with the input dimension. We implement Tree-RNN and Tree-Linearization for Transformer in a similar way, only adapting the source input word representing. We adopt a widely-used setting with 8 heads, 6 layers and the hidden dimension size of 512. Table 5 shows the results. As shown, the transformer results are indeed much better than RNN-based baseline. The BLEU scores show an average increase of 40 . 74  37 . 09 = 3 . 65 . In addition, we can see that syntax information can still give positive influences based on the transformer. The SAWR approach can also outperform the baseline system significantly. Particularly, we find that our SAWR approach is much more effective than the Tree-RNN and Tree-Linearization approaches. The results further demonstrate the effectiveness of SAWRs in syntax integration for NMT. By explicitly expressing the structural connections between words and phrases, syntax trees been demonstrated helpful in SMT (Liu et al., 2006; Cowan et al., 2006; Marton and Resnik, 2008; Xie et al., 2011; Li et al., 2013; Williams et al., 2016). Although the representative Seq2Seq NMT models are able to capture latent long-distance relations by using neural network structures such GRU and LSTM (Sutskever et al., 2014; Wu et al., 2016), recent studies show that explicitly integrating syntax trees into NMT models can bring further gains (Sennrich and Haddow, 2016; Shi et al., 2016; Zhou et al., 2017a; Wu et al., 2017a; Aharoni and Goldberg, 2017). Under the NMT setting, the exploration of syntax trees could be more flex-ible, because of the strong capabilities of neural network in representing arbitrary structures. Recursive neural networks based on LSTM or GRU have been one natural method to model syntax trees (Zhu et al., 2015; Tai et al., 2015; Li et al., 2015; Zhang et al., 2016; Teng and Zhang, 2016; Miwa and Bansal, 2016; Kokkinos and Potami-anos, 2017), which are capable of representing the entire trees globally. Eriguchi et al. (2016) present the first work to apply a bottom-up Tree-LSTM for NMT. The major drawback is that its bottom-up composing strategy is insufficient for bottom nodes. Thus bi-directional extensions have been suggested (Chen et al., 2017a; Yang et al., 2017). Since Tree-RNN suffers serious inefficiency problem, Li et al. (2017) suggest a Tree-Linearization alternative, which converts constituent trees into a sequence of symbols mixed with words and syntactic tags. The method is as effective as Tree-RNN approaches yet more effective. Noticeably, all these studies focus on constituent trees. There have been several studies for NMT using dependency syntax. Hashimoto and Tsuruoka (2017) propose to combine the head information with sequential words together as source encoder inputs, where their input trees are latent dependency graphs. Recently, there are several studies by using convolutional neural structures to represent source dependency trees, where tree nodes are modeled individually (Chen et al., 2017b; Bastings et al., 2017). Wu et al. (2017b) build a syntax enhanced encoder by multiple Bi-RNNs over several different word sequences based on different traversing orders over dependency trees, i.e., the original sequential order and several tree-based orders. All these methods require certain extra efforts to encode the source dependency syntax over a baseline Seq2Seq NMT. We proposed a novel syntax integration method, SAWR, to incorporate source dependency-based syntax for NMT. It encodes dependency syntax implicitly, not requiring discrete syntax trees as inputs. Experiments showed that the method can bring significantly better performances for both Chinese-English and English-Vietnamese translation tasks. In addition, we compared the method with two approaches based on Tree-RNN and Tree-Linearization, which has been previously exploited for syntax integration, finding that our method is more effective and meanwhile very efficient. We conducted several experimental analyses to study our proposed methods deeper. We thank all anonymous reviewers for their valuable comments. We thank Huadong Chen, Haoran Wei and Zaixiang Zheng for their help in implementing baseline neural machine translation models. This work is supported by National Natural Science Foundation of China (NSFC) grants 61525205, U1836222, and 61672211.
abstain abstain objective abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain other abstain abstain result result abstain abstain abstain abstain method abstain abstain abstain abstain method method result abstain result other abstain abstain abstain abstain abstain other method abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method method method abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain other abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain other other other other other other other other other other other other other other other objective abstain abstain result objective other other other