This paper proposes a simple and effective approach to address the problem of posterior collapse in conditional variational autoencoders (CVAEs). It thus improves performance of machine translation models that use noisy or monolingual data, as well as in conventional settings. Extending Transformer and conditional VAEs, our proposed latent variable model measurably prevents posterior collapse by (1) using a modified evidence lower bound (ELBO) objective which promotes mutual information between the latent variable and the target, and (2) guiding the latent variable with an auxiliary bag-of-words prediction task. As a result, the proposed model yields improved translation quality compared to existing variational NMT models on WMT Ro  En and De  En. With latent variables being effectively utilized, our model demonstrates improved robustness over non-latent Transformer in handling uncertainty: exploiting noisy source-side monolingual data (up to +3.2 BLEU), and training with weakly aligned web-mined parallel data (up to +4.7 BLEU). The conditional variational autoencoder (CVAE; Sohn et al., 2015) is a conditional generative model for structured prediction tasks like machine translation. This model, learned by variational Bayesian methods (Kingma and Welling, 2014), can capture global signal about the target in its latent variables. Unfortunately, variational inference for text generation often yields models that ignore their latent variables (Bowman et al., 2016), a phenomenon called posterior collapse . In this paper, we introduce a new loss function for CVAEs that counteracts posterior collapse, motivated by our analysis of CVAE's evidence lower bound objective (ELBO). Our analysis (  2) reveals that optimizing ELBO's second term not only brings the variational posterior approximation closer to the prior, but also decreases mutual information between latent variables and observed data. Based on this insight, we modify CVAE's ELBO in two ways (  3): (1) We explicitly add a principled mutual information term back into the training objective, and (2) we use a factorized decoder (Chen et al., 2017), which also predicts the target bag-of-words as an auxiliary decoding distribution to regularize our latent variables. Our objective is effective even without KullbackLeibler term (KL) annealing (Bowman et al., 2016), a strategy for iteratively altering ELBO over the course of training to avoid posterior collapse. In applying our method to neural machine translation (NMT; Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014), we find that we have measurably mitigated posterior collapse. The latent variables are not ignored, even in the presence of a powerful Transformer decoder. By addressing this problem, the resulting NMT model has improved robustness and performance in low-resource scenarios. Noisy data like those scraped from the Internet (Smith et al., 2013; Michel and Neubig, 2018) present a challenge for NMT (Khayrallah and Koehn, 2018; Ott et al., 2018a); we are measurably more able to model this extrinsic uncertainty than the (non-latent) Transformer (Vaswani et al., 2017) or existing variational NMT with the CVAE architecture (Zhang et al., 2016). Finally, we extend the model to semi-supervised learning (Cheng et al., 2016) to more effectively learn from monolingual data. In summary, our conditional text generation model overcomes posterior collapse by promoting mutual information. It can easily and successfully integrate noisy and monolingual data, and it does this without the cost of lower BLEU score than non-latent NMT in typical settings. Here we review the standard framework for neural MT. Next, we connect this to the conditional variational autoencoder, a model with latent random variables whose distributions are learned by black-box variational Bayesian inference. Finally, we analyze the CVAE's objective to explain why these models will ignore their latent variables (posterior collapse). Problem instances in machine translation are pairs of sequences ( x (cid:44) [ x 1 , . . . , x m ] , y (cid:44) [ y 1 , . . . , y n ]) , where x and y represent the source and target sentences, respectively. Conventionally, a neural machine translation model is a parameterized conditional distribution whose likelihood factors in an autoregressive fashion: p  ( y | x ) = n (cid:89) t =1 p  ( y t | x , y <t ) . The dominant translation paradigm first represents the source sentence as a sequence of contextual-ized vectors (using the encoder ), then decodes this representation into a target hypothesis according to Equation 1. The parameters  are learned by optimizing the log-likelihood of training pairs with stochastic gradient methods (Bottou and Cun, 2004; Kingma and Ba, 2015). Decoding is deterministic, using an efficient approximate search like beam search (Tillmann and Ney, 2003). The Transformer architecture with multi-head attention has become the state of the art for NMT (Vaswani et al., 2017). Our NMT approach extends the conditional variational autoencoder (Sohn et al., 2015), which we identify as a generalization of Variational NMT (Zhang et al., 2016). It introduces a latent random variable z into the standard NMT conditional distribution from Equation 1: 1,2 p  ( y | x ) = (cid:90) z p  ( y | z , x ) (cid:124) (cid:123)(cid:122) (cid:125) decoder  p  ( z | x ) (cid:124) (cid:123)(cid:122) (cid:125) encoder d z . For a given source sentence x , first a latent variable z is sampled from the encoder, then the target sen-1 tence y is generated by the decoder: z  p  ( z | x ) , y  p  ( y | z , x ) . 3 It is intractable to marginalize Equation 2 over z . Instead, the CVAE training objective is a variational lower bound (the ELBO) of the conditional log-likelihood. It relies on a parametric approximation of the model posterior: q  ( z | x , y ) . The variational family we choose for q is a neural network whose parameters  are shared (i.e., amortized) across the dataset. The ELBO lower-bounds the log-likelihood, as can be proven with Jensen's inequality. Its form is: LCVAE = E q  ( z | x , y ) [log p  ( y | x , z )]  DKL ( q  ( z | x , y ) (cid:107) p  ( z | x )) , (3) where DKL represents the KullbackLeibler divergence between two distributions. We use amortized variational inference to simultaneously perform learning and approximate posterior inference, updating both  and  with stochastic gradient methods. Improving  raises the lower bound, and improving  keeps the bound tight with respect to the model conditional log-likelihood. The same argument pertains to the joint maximization interpretation of the expectationmaximization (EM) algorithm (Neal and Hinton, 1998). (Our optimization is a variational generalization of EM.) 2.3 Posterior Collapse Despite their success when applied to computer vision tasks, variational autoencoders in natural language generation suffer from posterior collapse , where the learnt latent code is ignored by a strong autoregressive decoder. This presents a challenge to conditional language generation tasks in NLP like machine translation. The phenomenon can be explained mathematically by an analysis of the ELBO objective, as well as from the perspective of a powerful decoder that can model the true distribution without needing the latent code. We consider both in this subsection. ELBO surgery Recall that the computed objective approximates the objective on the true data distribution p D , using a finite number of samples 3 The sense of encoder in the context of variational autoencoders differs from the typical sense in neural machine translation, such that the NMT encoder is a component of both the VAE's encoder and decoder. We can separate these by computing a second, deterministic latent variable h from x to represent the NMT encoder outputs, used by both the VAE encoder and the NMT/VAE decoder. L = E p D ( x , y ) [ LCVAE (  ,  ; x , y )] . (4) We can factor the KL term of Equation 3 (omitting parameter subscripts) as: E p D ( x , y ) [DKL ( q ( z | x , y ) (cid:107) p ( z | x ))] = H( x , y )  H( x , y | z ) (cid:124) (cid:123)(cid:122) (cid:125) (cid:44) I q ( z ; x , y ) + E q ( z ) log q ( z ) p ( z ) (cid:124) (cid:123)(cid:122) (cid:125) (cid:44) DKL ( q  ( z ) (cid:107) p ( z )) , (5) which we prove in Appendix A, following (Hoff-man and Johnson, 2016). As both the resulting mutual information and KL terms are non-negative (Cover and Thomas, 2006), the global minimum of Equation 5 is I q  ( z ; x , y ) = DKL ( q  ( z ) (cid:107) p ( z )) = 0 . Unfortunately, at this point, the consequence of the optimization is that the latent variable z is conditionally independent of the data ( x , y ) . A powerful decoder Revisiting Equation 3, we see that the decoder is conditioned on both the stochastic latent variable z and the source text x . A sufficiently high-capacity autoregressive decoder can model the conditional density directly, ignoring the latent variable and reducing inference to Equation 1. The KL term can then be reduced to its minimum (0) by equating the posterior to the prior. To prevent this, some work weakens the decoder in various ways. This is a challenge, because NMT requires a powerful decoder such as Transformer with direct attention to the encoder. We modify our training objective to explicitly retain mutual information between the latent variable z and the observation ( x , y ) . Further, we use an auxiliary decoder that only uses the latent variable, not the encoder states. We combine it with the existing decoder as a mixture of softmaxes (Yang et al., 2018a). The model is trained with amortized variational inference. When source-language monolingual text is available, we augment our modified CVAE objective with a similarly modified (non-conditional) VAE objective. The training and inference strategy is summarized in Figure 1. 3.1 Adding I q  ( z ; x , y ) to ELBO To combat the optimization dilemma from Equation 5 (namely, that the objective discourages mutual information between the latent variable and the data), we explicitly add the mutual information term to the CVAE's ELBO and obtain a new training objective: LMICVAE = LCVAE + I q  ( z ; x , y ) = E q  ( z | x , y ) log p ( y | x , z )  DKL ( q  ( z ) (cid:107) p ( z )) (6) The new training objective LMICVAE aims to match the aggregated approximate posterior distribution of the latent variable q  ( z ) (Hoffman and Johnson, 2016) to the aggregated-posterior prior distribution p  ( z ) . 4 4 It can be seen as extending InfoVAE (Zhao et al., 2019) to conditional generative models, where we have overcome 3.2 Guiding z to Encode Global Information Several existing approaches weaken the decoder : limiting its capacity to encourage latent variables to be utilized (Bowman et al., 2016; Gulrajani et al., 2017). Here we propose a different approach: explicitly guiding the information encoded in z without reducing the decoder's capacity. The decision to weaken the decoder can be understood in the context of Bits-Back Coding theory (Chen et al., 2017), which suggests that at optimality the decoder will model whatever it can locally, and only the residual will be encoded in the latent variable z . A consequence is that explicit information placement can give more powerful latent representations. Inspired by this Bits-Back perspective, we add a global auxiliary loss for z to encode information which cannot be modelled locally by the autoregressive decoder (cid:81) t p  ( y t | x , y <t , z ) . We use bag-of-words (BoW) prediction as the auxiliary loss. It encodes global information while having a non-autoregressive factorization: (cid:81) t p  ( y t | z ) . (We choose not to condition it on the source sentence x .) Further, it requires no additional annotated data. The auxiliary decoder complements the autoregressive decoder (which is locally factor-ized), interpolating predictions at the softmax layer, i.e. p ( y t | x , y <t , z ) is a mixture of softmaxes (Yang et al., 2018b): p ( y t |  ) = (1   )  p  ( y t | x , y <t , z ) +   p  ( y t | z ) , (7) with mixing parameter  . Our model uses discrete latent variables. These are used to select a latent embedding, which is concatenated to the decoder state. Inference Network We use discrete latent variables with reparameterization via GumbelSoftmax (Jang et al., 2017; Maddison et al., 2017) to allow backpropagation through discrete sampling. Unlike the multivariate Gaussian distribution commonly used in VAE and CVAE, our parameterization can explicitly account for multiple the mismatch between the (joint) data distribution p D ( x , y ) and the (conditional) likelihood objective p  ( y | x ) . modes in the data. (See Rezende and Mohamed (2015) for a perspective on the value of multimodal distributions over latent variables.) To make our model more general, we introduce a set of discrete latent variables z = { z 1 , . . . , z K } which are independently sampled from their own inference networks  k . Specifically, each  k computes scaled dot product attention with encoder outputs h  R d using latent code embedding e k : C k = Attention (cid:16) e k W k , hW h , hW h (cid:17) = Softmax (cid:18) e k W k ( hW h ) (cid:62)  d (cid:19) hW h . We can now sample z k by the Gumbel-Softmax reparameterization trick (Maddison et al., 2017; Jang et al., 2017): z k  GumbelSoftmax ( C k ) = Softmax (cid:18) C k + g  (cid:19) , where g =  log(  log( u )) , u  Uniform is the Gumbel noise and  is a fixed temperature. (We use  = 1 in this paper.) At inference time, we use a discrete version by directly sampling from the latent variable distribution. BoW Auxiliary Decoder Given an inferred sample z   k ( h ) , the BoW decoder predicts all tokens at once without considering their order. We compute the cross-entropy loss for the predicted tokens over the output vocabulary space V : L BoW = | V | (cid:88) i =1 p i log  p  ( y i | z ) , | V | (cid:88) i =1 p i = 1 . We take the (unnormalized) empirical distribution  p i to be a token's frequency within a sentence normalized by its total frequency within a mini-batch, mitigating the effect of frequent (stop) words. This is then normalized over the sentence to sum to 1, giving values p i . The model distribution  p  is computed by conditioning on the latent code only, without direct attention to encoder outputs. We use scaled dot-product attention between the latent embeddings and the target embeddings (each of dimensionality d , represented as a matrix EV ): p  ( y i | z ) = Softmax (cid:18) e ( z ) E (cid:124) V  d (cid:19) i . For training with parallel data, we optimize LMICVAE . We draw samples z from the approximate posterior q  ( z | x , y ) parameterized by the inference network, then feed the samples to both the autoregressive and auxiliary (BoW) decoders to get a Monte Carlo estimate of the gradient. Estimating aggregated distributions We estimate p  ( z ) and q  ( z ) over each minibatch, following Zhao et al. (2018). Semi-supervised learning We apply the same modification to VAE's ELBO, following Zhao et al. (2019). For jointly training with source-side monolingual data, we add I q  ( z ; x ) to the ELBO, and for target-side monolingual data, we add I q  ( z ; y ) . 5 The joint objective sums the modified CVAE and VAE objectives: L Mono = log p ( x | z ) + DKL (cid:32) 1 LL (cid:88) (cid:96) =1 q  (cid:16) z ( (cid:96) ) (cid:12)(cid:12)(cid:12) x ( (cid:96) ) (cid:17) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) 1 LL (cid:88) (cid:96) =1 p (cid:16) z ( (cid:96) ) (cid:17)(cid:33) (13) L Joint = LMICVAE + L Mono , (14) where L is the number of monolingual examples. Algorithm 1 describes the overall training strategy. Here we present empirical results on the Transformer architecture. We evaluate our model on four standard datasets and compare against three baselines. We use four measures to quantify posterior collapse, then examine translation quality (BLEU score) in standard fully supervised settings, a semi-supervised setting, and a fully supervised setting with noisy source text. Hyperparameters, regularization choices, and subword vocabulary information can be found in  5.3. The results show that we have effectively addressed posterior collapse: latent variables are no longer ignored despite the presence of a powerful decoder. As a result, we outperform both the standard Transformer and the Transformer-based variational NMT approach, when using noisy data or source-language monolingual data. First, we evaluate our models on a standard high-resource and low-resource benchmark dataset from WMT. Second, we focus on situations where noisy or monolingual data is available. We note that low-resource scenarios and noisy data are two representative challenges in MT (Lopez and Post, 2013). WMT16 RomanianEnglish We use data from the WMT16 news translation shared task. We use the same BPE-preprocessed (Sennrich et al., 2016b) train, dev and test splits as in Gu et al. (2018) with 608k sentence pairs for training. This dataset pairs web-scraped text from Reddit with professional translations. We use 30k subword units built jointly from source and target sentences and only keep sentences with less than 100 tokens. For training, there are 34,380 sentence pairs for EnglishFrench and 17,616 sentence pairs for FrenchEnglish (Michel and Neubig, 2018). We also used 18,676 monolingual sentences per language from the same data source (Reddit). We compare our model to three baselines: without latent variables. VNMT A CVAE model with Gaussian distribution as proposed in Variational NMT by Zhang et al. (2016), which we reimplement using Transformer. (Zhang et al. (2016) use a GRU-based recurrent model.) DCVAE A CVAE model with the same discrete latent variable parameterization as ours but without the new objective (i.e., the mutual information term and bag-of-words regularizer). All of our models build on Transformer. For WMT14 DeEn and WMT16 RoEn, we use the base configuration (Vaswani et al., 2017): 6 blocks, with 512-dimensional embedding, 2048-dimensional feed-forward network, and 8 attention heads. For FLoRes (low-resource) and MTNT (low-resource and noisy), we use a smaller Transformer: 4 layers, 256-dimensional embedding, 1024-dimensional inner layers, and 4 attention heads. Input and output embeddings are shared between the inference network and decoder. We use T = 4 categorical latent variables of dimension 16 (found by grid search on the dev set). Auxiliary bag-of-words predictions are combined with the decoder prediction with  = 0 . 1 . We optimize using Adam (Kingma and Ba, 2015) with  1 = 0 . 9 ,  2 = 0 . 98 , (cid:15) = 1 E8 , weight decay of 0.001, and the warmup and learning rate schedule of Ott et al. (2018b). All models are trained on 8 NVIDIA V100 GPUs with 32K tokens per mini-batch. We train WMT14 DeEn with 200k updates and others with 100k updates. We do not use early stopping. We employ joint BPE vocabularies. The sizes are 32k for EnDe and EnRo; 30k for FrEn; and 3k for SiEn. We also use a word dropout rate of 0.4 during training of all models, which is complementary to our approach. We found the default initialization in the FAIRSEQNMT toolkit was effective; we did not need to explore several initializations to avoid degenerate models. We compare our model to a standard DCVAE lacking the new objective. We report four metrics of posterior collapse on the validation set of WMT RoEn: 1. KullbackLeibler divergence (KL). 2. Mutual information between the latent variable and the source: I q  ( z ; x ) 3. Mutual information between the latent variable and the target: I q  ( z ; y ) . 4. Negative conditional log-likelihood (NLL) per token. Table 1 shows that when using standard DCVAE ELBO, even with the common practice of KL annealing (KLA), both the KL loss and mutual information settle to almost 0 which is consistent with the analysis in Equation 5. We also plot the progression of DKL , I q  ( z ; x ) , and I q  ( z ; y ) during training in Figure 2. The posterior collapse of the baseline model is apparent: both DKL mutual information terms drop to 0 at the beginning of training as a result ELBO's design. On the other hand, our model, without using any annealing schedule, effectively increases mutual information and prevents KL loss from settling to a degenerate solution early on. We report corpus-level BLEU (Papineni et al., 2002) 6 on the test sets where the translations are generated by sampling each z k with soft-assignment (vs. argmax). Supervised Learning on Parallel Data First, we evaluate our model's performance when trained with parallel data on standard WMT datasets. Table 2 shows that our model consistently outperforms both VNMT and DCVAE modelswhich 6 We use detokenized SacreBLEU (Post, 2018). Data Leveraging monolingual data is a common practice to improve low resource NMT. One popular approach uses target-side monolingual data through backtranslation as a data augmentation, but how to effectively leverage source-side monolingual data is an open challenge (Sennrich et al., Model FrEn EnFr Non-latent 26.7 24.8 DCVAE 26.4 26.1 + source mono 27.3 26.4 Our model 28.6 26.3 + source mono 29.8 26.7 Table 3: Translation performance (BLEU) of utilizing source-side monolingual data. 2016a; Zhang and Zong, 2016; Wu et al., 2019). We use the joint training objective described in Equation 14. To have a fair comparison, we also extend VNMT and DCVAE with the same joint training algorithm, i.e., the newly added monolingual data is used to train their corresponding sequence encoder and inference network with standard VAE ELBO. That is, the only difference is that our model was trained to promote mutual information I q  ( z , x ) and I q  ( z , y ) . As shown in Table 3, by doing so the proposed model brings larger gains during semi-supervised learning with source-side monolingual data. Robustness to Noisy Data While high-quality parallel data is scarce for low-resource language pairs, weakly aligned sentence pairs can be mined from massive unpaired data such as Paracrawl. 7 We evaluate our model's performance when augmenting the training set with increasingly noisy parallel data filtered by Zipporah (Xu and Koehn, 2017). Because VNMT and DCVAE underperform our proposal in previous experiments, we omit them from this experiment. Figure 3 shows the results in the SinhalaEnglish direction. Our model always outperforms standard Transformer, which struggles as more (and noisier) data is added. The gap grows from +1.2 to +4.7 BLEU. Ablation Study How do the different ingredients of our proposed approach contribute to preventing posterior collapse and improving translation quality? We explore two variants of the proposed model: 1) modified ELBO only: only adding mutual information term to the training objective, while without gradients from L BoW , 2) BoW only: which is equivalent to DCVAE combined with BoW decoder. First, we perform the same collapse metrics evaluation as in Table 1. Figure 2(B) suggests that by explicitly adding mutual information term back to the training objective, both I q  ( z ; x ) and I q  ( z ; y ) are effectively raised, while the remaining aggregated KL term is still optimized to zero. Such behavior is consistent with the analysis revealed 7 https://paracrawl.eu/ Model DeEn (3.9M) RoEn (608K) BoW and LMICVAE 31.4 34.8 BoW only 31.1 34.2 Table 4: Ablation study on translation quality (BLEU). in Equation 5. On the other hand, regularizing z with the BoW decoder only, shown in Figure 2(C), is very effective in preventing KL vanishing as well as increasing mutual information. When two approaches are combined, as was shown in Figure 2(A), the model retains higher mutual information for both I q  ( z ; x ) and I q  ( z ; y ) . Next, we see whether the difference in mutual information yields different translation quality. We compare two models: BoW only (Figure 2(C)) and both (Figure 2(A)), on WMT14 DeEn and WMT16 RoEn test sets. Table 4 shows the difference matters more in a low-data regime. Analysis of Outputs Delving into model predictions helps us understand how our model outperforms the others. We examined erroneous 1-best predictions on the RoEn data. We provide salient examples of phenomena we identified in Table 5. (Naturally, as the RoEn score differences are not dramatic, the predictions are largely similar.) Several examples support the fact that our model has more fluent and accurate translations than the baseline or VNMT. VNMT often struggles by introducing disfluent words, and both VNMT and Transformer select justifiable but incorrect words. For instance, in our second example, the gender and animacy of the possessor are not specified in Romanian. Our model selects a more plausible pronoun for this context. Analysis of Latent Variables Finally, we probe whether different latent variables encode different information. We random sample 100 sentences from two test sets of distinct domains, MTNT (Reddit comments) and WMT (news) with 50 sentences each. We plot the t -SNE projection of their corresponding samples z k inferred from  k , k = 1 , 2 , 3 , 4 respectively. Figure 4 suggests that different latent variables learn to organize the data in different manners, but there was no clear signal that any of them exclusively specialize in encoding a domain label. We leave a thorough analysis of Source : ma intristeaza foarte tare . Unlike most prior work in (conditional) text generation, we tackle posterior collapse without requiring an annealing schedule (Bowman et al., 2016; Snderby et al., 2016; Kim et al., 2018), a weakened decoder (Gulrajani et al., 2017), or a restricted variational family (Razavi et al., 2019). Unlike Ma et al. (2018), who also employ bag-of-words as an NMT objective, our BoW decoder only sees the latent variable z , not the encoder states. Conversely, unlike Weng et al. (2017), our generative decoder has access to both the latent variable and the encoder states; bag-of-words prediction is handled by separate parameters. VNMT (Zhang et al., 2016) applies CVAE with Gaussian priors to conditional text generation. VRNMT (Su et al., 2018) extends VNMT, modeling the translation process in greater granularity. Both needed manually designed annealing schedules to increase KL loss and avoid posterior collapse. Discrete latent variables have been applied to NMT (Kaiser et al., 2017; Gu et al., 2018; Shen et al., 2019), without variational inference or addressing posterior collapse. Approaches to stop posterior collapse include aggressively trained inference networks (He et al., 2019), skip connections (Dieng et al., 2019), and expressive priors (Tomczak and Welling, 2018; Razavi et al., 2019). Unlike our conditional approach, Shah and Barber (2018) jointly model the source and target text in a generative fashion. Their EM-based inference is more computationally expensive than our amortized variational inference. Eikema and Aziz (2019) also present a generative (joint) model relying on autoencoding; they condition the source text x on the latent variable z . Finally, Schulz et al. (2018), like us, value mutual information between the data and the latent variable. While they motivate KL annealing using mutual information, we show that the annealing is unnecessary. We have presented a conditional generative model with latent variables whose distribution is learned with variation inference, then evaluated it in machine translation. Our approach does not require an annealing schedule or a hamstrung decoder to avoid posterior collapse. Instead, by providing a new analysis of the conditional VAE objective to improve it in a principled way and incorporating an auxiliary decoding objective, we measurably prevented posterior collapse. As a result, our model has outperformed previous variational NMT models in terms of translation quality, and is comparable to non-latent Transformer on standard WMT Ro  En and De  En datasets. Furthermore, the proposed method has improved robustness in dealing with uncertainty in data, including exploiting source-side monolingual data as well as training with noisy parallel data. We thank Alexandra DeLucia, Chu-Cheng Lin, Hongyuan Mei, Kenton Murray, Guanghui Qin, and Joao Sedoc (alphabetically) for remarks on the exposition.
objective abstain objective abstain objective abstain abstain abstain objective result abstain abstain result abstain abstain method abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method abstain abstain method abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain other abstain abstain abstain abstain method method abstain method abstain other method abstain method abstain abstain abstain method abstain method abstain abstain method method abstain method method abstain abstain method method abstain method method method other abstain result abstain abstain abstain result result abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain other other other other other abstain abstain other abstain objective method abstain objective result abstain other