Transformer architecture achieves great success in abundant natural language processing tasks. The over-parameterization of the Transformer model has motivated plenty of works to alleviate its overfitting for superior performances. With some explorations, we find simple techniques such as dropout, can greatly boost model performance with a careful design. Therefore, in this paper, we integrate different dropout techniques into the training of Transformer models. Specifically, we propose an approach named UniDrop to unite three different dropout techniques from fine-grain to coarse-grain, i.e., feature dropout, structure dropout, and data dropout. Theoretically, we demonstrate that these three dropouts play different roles from regularization perspectives. Empirically, we conduct experiments on both neural machine translation and text classification benchmark datasets. Extensive results indicate that Transformer with UniDrop can achieve around 1 . 5 BLEU improvement on IWSLT14 translation tasks, and better accuracy for the classification even using strong pre-trained RoBERTa as backbone. In recent years, Transformer (Vaswani et al., 2017) has been the dominant structure in natural language processing (NLP), such as neural machine translation (Vaswani et al., 2017), language modeling (Dai et al., 2019) and text classification (Devlin et al., 2019; Liu et al., 2019). To further improve the model performance, there has been much effort in designing better architectures or introducing external knowledge into Transformer models (Wu et al., 2019; Lu et al., 2019; Kitaev et al., 2020; Ahmed et al., 2017; Hashemi et al., 2020), which increases computational costs or requires extra resources. a crucial problem for Transformer. Regularization methods such as weight decay (Krogh and Hertz, 1992), data augmentation (Sennrich et al., 2016a), dropout (Srivastava et al., 2014), parameter sharing (Dehghani et al., 2018; Xia et al., 2019) are all widely adopted to address overfitting. Among these regularization approaches, dropout (Srivas-tava et al., 2014), which randomly drops out some hidden units during training, is the most popular one and various dropout techniques have been proposed for Transformer. For example, Fan et al. (2020a) propose LayerDrop , a random structured dropout, to drop certain layers of Transformer during training. Zhou et al. (2020) alternatively propose DropHead as a structured dropout method for regularizing the multi-head attention mechanism. Both of them achieved promising performances. One great advantage of dropout is that it is free of additional computational costs and resource requirements. Hence we ask one question: can we achieve stronger or even state-of-the-art (SOTA) results only relying on various dropout techniques instead of extra model architecture design or knowledge enhancement? To this end, in this paper, we propose UniDrop to integrate three different-level dropout techniques from fine-grain to coarse-grain, feature dropout , structure dropout , and data dropout , into Transformer models. Feature dropout is the conventional dropout (Srivastava et al., 2014) that we introduced before, which is widely applied on hidden representations of networks. Structure dropout is a coarse-grained control and aims to randomly drop some entire substructures or components from the whole model. In this work, we adopt the aforementioned LayerDrop (Fan et al., 2020a) as our structure dropout. Different from the previous two dropout methods, data dropout (Iyyer et al., 2015) is performed on the input data level, which serves as a data augmentation method by randomly dropping out some tokens in an input sequence. (a) Transformer architecture. (b) Structure and overview of feature dropout. We first theoretically analyze different regularization roles played by the three dropout techniques, and we show they can improve the generalization ability from different aspects. Then, we provide empirical evaluations of the UniDrop approach. We conduct experiments on neural machine translation with 8 translation datasets, and text classification task with 8 benchmark datasets. On both sequence generation and classification tasks, experimental results show that the three dropouts in UniDrop can jointly improve the performance of Transformer. The contributions of this paper can be summarized as follows:  We introduce UniDrop , which unites three different dropout techniques into a robust one for Transformer, to jointly improve the performance of Transformer without additional computational cost and prior knowledge. We theoretically demonstrate that the three dropouts, i.e., feature dropout, structure dropout, and data dropout play different roles in preventing Transformer from overfitting and improving the robustness of the model. Extensive results indicate that Transformer models with UniDrop can achieve strong or even SOTA performances on sequence generation and classification tasks. Specifically, around 1 . 5 BLEU improvement on IWSLT14 translation tasks, and better accuracy for classification even using strong pre-trained model RoBERTa as backbone. Feature dropout (FD) and structure dropout (SD) are highly coupled with model architecture. Therefore, we briefly recap Transformer and refer the readers to Vaswani et al. (2017) for details. As shown in Figure 1a, Transformer is stacked by several identical blocks, and each block contains two sub-layers, which are multi-head self-attention layer and position-wise fully connected feed-forward layer. Each sub-layer is followed by an AddNorm operation that is a residual connection Add (He et al., 2016) and a layer normalization LN (Ba et al., 2016). Multi-head Attention sub-layer consists of multiple parallel attention heads, and each head maps the query Q and a set of key-value pairs K , V to an output through a scale dot-product attention: Attn( Q , K , V ) = softmax( QK (cid:62)  d k ) V , (1) where d k is the dimension of query and key, and 1  d k is a scaling factor. The outputs of these heads are then concatenated and projected again to result in the final values. Position-wise Feed-Forward sub-layer applies two linear transformations with an inner ReLU (Nair and Hinton, 2010) activation: FFN ( x ) = max(0 , xW 1 + b 1 ) W 2 + b 2 , (2) where W and b are parameters. In this section, we first introduce the details of the three different levels of dropout techniques we study, feature dropout, structure dropout and data dropout. Then we provide the theoretical analysis of these dropout methods on the regularization perspectives. Finally, we present our proposed UniDrop approach for training Transformer. The feature dropout (FD), as a well-known regularization method, is proposed by Srivastava et al. (2014), which is to randomly suppress neurons of neural networks during training by setting them to 0 with a pre-defined probability p . In practice, dropout is applied to the output of each sub-layer by default. Besides, Transformer also contains two specific feature dropouts for multi-head attention and activation layer of feedforward network. In this work, we also explore their effects on the performance of Transformer. FD-1 (attention dropout): according to Equation (1), we can obtain attention weight matrix A = QK (cid:62) towards value sequence V . Our FD-1 is applied to the attention weight A . FD-2 (activation dropout): FD-2 is employed after the activation function between the two linear transformations of FFN sub-layer. In addition to the above FDs for Transformer, we still find the risk of overfitting in pre-experiments. Therefore, we further introduce another two feature dropouts into the model architecture:  FD-3 (query, key, value dropout): FD-1 is used to improve generalization of multi-head attention. However, it is directly applied to the attention weights A , where drop value A ( i, j ) means ignore the relation between token i and token j , thus a larger FD-1 means a larger risk of losing some critical information from sequence positions. To alleviate this potential risk, we add dropout to query, key, and value before the calculation of attention. FD-4 (output dropout): we also apply dropout to the output features before linear transformation for softmax classification. Specifically, when dealing with sequence-to-sequence tasks such as machine translation, we add FD-4 to the output features of the last layer in the Transformer decoder, otherwise the last layer of the Transformer encoder. The positions of each feature dropout applied in Transformer 1 are shown in Figure 1b. There are three structure dropouts, respectively LayerDrop (Fan et al., 2020a), DropHead (Zhou et al., 2020) and HeadMask (Sun et al., 2020), which are specifically designed for Transformer. Some recent studies (Voita et al., 2019; Michel et al., 2019) show multi-head attention mechanism is dominated by a small portion of attention heads. To prevent domination and excessive coadaptation between different attention heads, Zhou et al. (2020) and Sun et al. (2020) respectively propose structured DropHead and HeadMask that drop certain entire heads during training. In contrast, LayerDrop (Fan et al., 2020a) is a higher-level and coarser-grained structure dropout. It drops some entire layers at training time and directly reduces the Transformer model size. In this work, we adopt LayerDrop as the structure dropout to incorporate it into our UniDrop . Data dropout aims to randomly remove some words in the sentence with a pre-defined probability. It is often used as a data augmentation technique (Wei and Zou, 2019; Xie et al., 2020). However, directly applying vanilla data dropout is hard to keep the original sequence for training, which leads to the risk of losing high-quality training samples. To address this issue, we propose a two-stage data dropout strategy . Specifically, given a sequence, with probability p k (a hyperparameter lies in (0 , 1) ), we keep the original sequence and do not apply data dropout. If data dropout is applied, for each token, with another probability p (another hyperparameter lies in (0 , 1) ), we will drop the token. In this section, we provide theoretical analysis for feature dropout, structure dropout and data dropout, to show their different regularization effects. We first re-formulate the three dropout methods. For some probability p and layer representation h  R d (i.e., h is the vector of outputs of some layer), we 1 We also explored other positions for feature dropout, but their performances are not so good (see Appendix A.3). randomly sample a scaling vector   R d with each independent coordinate as follows:  i = (cid:40)  1 with probability p p 1  p with probability 1-p . (3) Here, i indexes a coordinate of  , i  [1 , ..., d ] . Then feature dropout can be applied by computing h fd = ( 1 +  ) (cid:12) h, where (cid:12) denotes element-wised product and 1 = (1 , 1 ,    , 1) (cid:48) . Similar to Wei et al. (2020), we apply Taylor expansion to L and take expectation to  : E  L ( F ( h fd ( x ))) = E  L ( F (( 1 +  ) (cid:12) h ( x )))  L ( F ( h ( x )) + 1 2 E  (  (cid:12) h ( x )) TD 2 h L ( x )(  (cid:12) h ( x )) = L ( F ( h ( x )) + p 2(1  p ) d (cid:88) j =1 D 2 h j ,h j L ( x )  h j ( x ) 2 , (4) where D 2 h L is the Hessian matrix of loss with respect to hidden output h and D 2 h j ,h j L ( x ) is the j -th diagonal element of D 2 h L . We use F ( h fd ( x )) to denote the output of a model after dropping feature from a hidden layer and L to denote the loss function. Expect the original loss L ( F ( h ( x ))) , the above formula shows that feature dropout implicitly regularize the term (cid:80) dj =1 D 2 h j ,h j L ( x )  h j ( x ) 2 , which relates to the trace of the Hessian. For structure dropout, we use a 1-dim random scalar   R whose distribution is:  =  1 with probability p , and  = 0 with probability 1  p . The structure dropout is similarly applied by computing h sd = (1 +  )  h . For input data x  R m , here x is a sequence of tokens and m is the sequence length, we sample a random scaling vector   R m with independent random coordinates where each coordinate is identically distributed as  . The input data after drop data becomes x dd = ( 1 +  ) (cid:12) x . Similar to feature dropout, we can obtain that data dropout implicitly optimizes the regularized loss as follows: L ( F ( h ( x )))  p  x T  x L ( x ) + p  (cid:80) mj =1 D 2 x j ,x j L ( x )  x 2 j , and structure dropout implicitly optimizes the regularized loss: L ( F ( h ( x )))  p  h ( x ) T  h L ( x ) + p  (cid:80) mi,j =1 D 2 h i ,h j L ( x )  h i ( x ) h j ( x ) , where D 2 h i ,h j L ( x ) is the ( i, j ) -th element in Hessian matrix D 2 h L . Interpretation From the above analysis, we can conclude that feature dropout, structure dropout and data dropout regularize different terms of the Data Dropout Layer Dropout Feature Dropout  1    +1  +2 Figure 2: Different dropout components in UniDrop . model, and they can not be replaced by each other. (1) Because the hidden output will be normalized by layer normalization, the term h ( x ) T  h L ( x ) equals to zero according to Lemma 2.4 in Arora et al. (2019). Therefore, structure dropout implicitly regularizes the term (cid:80) mi,j =1 D 2 h i ,h j L ( x ) . Hence, structure dropout can regularize the whole elements of Hessian of the model with respect to hidden output, while feature dropout only regularizes the diagonal elements of the Hessian. Thus, integrating structure dropout and feature dropout can regularize every component of Hessian with emphasizing the diagonal elements of the Hessian. (2) Since x is also normalized, the term x T  x L ( x ) equals to zero according to Lemma 2.4 in Arora et al. (2019). Different from feature dropout and structure dropout, data dropout regularizes Hessian of loss with respect to input data. Regularizing Hessian matrix with respect to both input and hidden output can improve model robustness and hence the generalization ability. We put more details in Appendix A.1. From the above theoretical analysis, the three dropout techniques are performed in different ways to regularize the training of Transformer, each with unique property to improve the model generalization. Therefore, we introduce UniDrop to take the most of each dropout into Transformer. The overview of UniDrop is presented in Figure 2. To better view each dropout in a model forward pass, we only show a three layers of architecture in Figure 2, and each layer with one specific dropout technique. The data dropout is applied in the input layer by dropping out some word embeddings (e.g., embedding of word t i is dropped). In the middle layer, the feature dropout randomly drops several neurons in each word representations (e.g., the third neurons of word t i  1 is dropped). The last layer is directly dropped out through layer dropout 2 . We conduct experiments on both sequence generation and classification tasks, specifically, neural machine translation and text classification, to validate the effectiveness of UniDrop for Transformer. In this section, we introduce the detailed settings for the neural machine translation tasks and report the experimental results. We adopt the widely acknowledged IWSLT14 datasets 3 with multiple language pairs, including English  German (En  De), English  Romanian (En  Ro), English  Dutch (En  Nl), and English  Portuguese-Brazil (En  Pt-br), a total number of 8 translation tasks. Each dataset contains about 170k  190k translation data pairs. The datasets are processed by Moses toolkit 4 and byte-pair-encoding (BPE) (Sennrich et al., 2016b) is applied to obtain subword units. The detailed statistics of datasets are shown in Appendix A.2. We use the transformer_iwslt_de_en configuration 5 for all Transformer models. Specifically, the encoder and decoder both consist of 6 blocks. The source and target word embeddings are shared for each language pair. The dimensions of embedding and feed-forward sub-layer are respectively set to 512 and 1024 , the number of attention heads is 4 . The default dropout (not our four feature dropout) rate is 0 . 3 and weight decay is 0 . 0001 . All models are optimized with Adam (Kingma and Ba, 2015) and the learning rate schedule is same as in Vaswani et al. (2017). The weight of label smoothing (Pereyra et al., 2017) is set to 0 . 1 . For the Transformer models with our UniDrop , we set all feature dropout rates to 0 . 1 . The structure dropout LayerDrop is only applied to the decoder with rate 0 . 1 . For the data dropout, the sequence 2 Except the data dropout is only applied in the input layer, feature/structure dropout can be applied in each layer. keep rate p k and token dropout rate p are respectively 0 . 5 and 0 . 2 . The other settings are the same as the configuration of the baseline Transformer. To evaluate the model performance, we use beam search (Sutskever et al., 2014) algorithm to generate the translation results. The beam width is 5 and the length penalty is 1 . 0 . The evaluation metric is the tokenized BLEU (Papineni et al., 2002) score with multi-bleu.perl script 6 . We repeat each experiment three times with different seeds and report the average BLEU. Table 1 shows the BLEU results of the Transformer baselines and models with different dropouts. Compared with baselines, we can see that the dropouts FD, SD, or DD all bring some improvements 7 . This observation verifies the existence of overfitting in the Transformer. In contrast, our model Transformer+ UniDrop achieves the most improvements across all translation tasks, which demonstrates the effectiveness of UniDrop for the Transformer architecture. To further explore the effects of the three different grained dropouts in UniDrop , we conduct ablation studies and respectively remove the FD, SD, and DD from Transformer+ UniDrop . The results in Table 1 show that three ablated models obtain lower BLEU scores compared to the full model. This observation validates the necessity of them for UniDrop . Among all ablation versions, the Transformer-UniDrop w/o FD obtains the least improvements. It is reasonable because FD actually contains four feature dropouts on different positions, which can effectively prevent Transformer from overfitting. To show the superiority of UniDrop , we also compare the Transformer+ UniDrop with several existing works on the widely acknowledged benchmark IWSLT14 De  En translation. These works improve machine translation from different aspects, such as the training algorithm design (Wang et al., 2019b), model architecture design (Lu et al., 2019; Wu et al., 2019) and data augmentation (Gao et al., 2019). The detailed results are shown in Table 2. We can see that the Transformer model with our UniDrop outperforms all previous works and achieve state-of-the-art performance, with 36 . 88 6 https://github.com/moses-smt/ mosesdecoder/blob/master/scripts/generic/multi-bleu.perl 7 The dropout rates of model Transformer+FD, Trans-former+SD, Transformer+DD are tuned with IWSLT14 De  En dev set and respectively set to 0 . BLEU score. Especially, it surpasses the BERT-fused NMT model (Zhu et al., 2020), which incorporates the pre-trained language model BERT, by a non-trivial margin. We also show some comparisons on IWSLT14 En  De, Ro  En, and Nl  En translations, the results are shown in Table 3. According to the above results, UniDrop successfully unites the FD, SD, and DD, and finally improves the performance of Transformer on neural machine translation tasks, without any additional computation costs and resource requirements. We also conduct experiments on text classification tasks to further demonstrate the effectiveness of UniDrop for the Transformer models. We evaluate different methods on the text classification task based on 8 widely-studied datasets, which can be divided into two groups. The first group is from GLUE tasks (Wang et al., 2019a), and they are usually used to evaluate the performance of the large-scale pre-trained language models after fine-tuning. The second group is some typical text classification datasets that are widely used in previous works (Voorhees and Tice, 1999; Maas et al., 2011; Zhang et al., 2015). The statistics of all datasets are shown in Appendix A.2. We employ RoBERTa BASE (Liu et al., 2019) as the strong baseline and fine-tune it on the text classification datasets. Different from BERTBASE (Devlin et al., 2019), RoBERTa BASE is pre-trained with dynamic masking, full-sentences without NSP loss and a larger mini-batches. It has 12 blocks, and the dimensions of embedding and FFN are 768 and 3072 , the number of attention heads is 12 . When fine-tuning, we set the batch size to 32 and the max epoch to 30 . Adam is applied to optimize the models with a learning rate of 1e-5 and a warm-up step ratio of 0 . 1 . We employ the polynomial decay strategy to adjust the learning rate. The default dropout and weight decay are both set to 0 . 1 . When adding UniDrop to RoBERTa BASE , we empirically set feature dropout rate and LayerDrop rate to 0 . 1 . For data dropout, the sequence keep rate p k and token dropout rate p are respectively 0 . 5 and 0 . 1 . The other settings are the same as in the baseline RoBERTa BASE . We use the standard accuracy to evaluate different methods on text classification tasks. Table 4 and Table 5 respectively show the accuracy of different models on GLUE tasks and typical text classification datasets. Compared with the conventional BiLSTM and CNN based models, we can observe the pre-trained models, including ULMFiT, BERT, RoBERTa, achieve obvious improvements on most datasets. Benefiting from better training strategy, RoBERTa BASE outperforms BERTBASE and even BERTLARGE on GLUE tasks. We can see our proposed UniDrop further improve the performance RoBERTa BASE on both small-scale and large-scale datasets. Specifically, UniDrop brings about 0 . 4 improvements of accuracy on the typical text classification datasets from Table 5. In contrast, RoBERTa BASE + UniDrop achieves more improvements on GLUE tasks. The experimental results on the 8 text classification benchmark datasets consistently demonstrate the facilitation of UniDrop for Transformer. We show more results and ablation study on text classification task in Appendix A.5. In this section, we use IWSLT14 De  En translation as the analysis task to investigate the capability of UniDrop to avoid overfitting, as well as the effects of different dropout components and dropout To show the superiority of UniDrop to prevent Transformer from overfitting, we compare the dev loss during training of Transformer, Transformer with each dropout technique, Transformer+ UniDrop , and ablated models of Transformer+ UniDrop . Figure 3 shows loss curves of different models. We can observe that the standard Transformer is quickly overfitted during training, though it is equipped with a default dropout. In contrast, the feature dropout, structure dropout, and data dropout, as well as the combinations of any two dropouts (i.e., ablated models), greatly reduce the risk of overfitting to some extent. Among all compared models, our Transformer+ UniDrop achieves the lowest dev loss and shows great advantage to prevent Transformer from overfitting. Besides, we also find that the dev loss of Transformer+ UniDrop continuously falls until the end of the training. We stop it to keep training epochs of all models same for a fair comparison. In Appendix A.4, we also plot the curves of training loss for the above models, together with the dev loss, to make a better understanding of the regularization effects from these dropout techniques. In Table 1, we have presented some important ablation studies by removing FD, SD, or DD from UniDrop . The consistent decline of BLEU scores demonstrates their effectiveness. Besides, we further investigate the effects of the two existing feature dropouts FD-1, FD-2, two new feature dropouts FD-3, FD-4, and our proposed two-stage  1 0 0.1 0.2 0.3 0.4 FD dev 36.94 38.00 37.30 35.29 35.68 36.88 36.33 34.71 SD dev 37.91 38.00 37.72 36.70 36.70 36.88 36.59 35.65 DD dev 37.52 37.80 38.00 37.87 37.69 36.38 36.74 36.88 36.75 36.55 BLEU s c o r e 34.5 35.5 36.5 37.5 38.5 0 0.1 0.2 0.3 Dev Test BLEU s c o r e 34.5 35.5 36.5 37.5 38.5 0 0.1 0.2 0.3 Dev Test BLEU s c o r e 34.5 35.5 36.5 37.5 38.5 0 0.1 0.2 0.3 0.4 Dev Test From Table 6, we can see the four ablation models removing FDs underperform the full model Transformer+ UniDrop , which means they can work together to prevent Transformer from overfitting. In multi-head attention module, FD-3 brings more BLUE improvement than FD-1. This comparison shows the insufficiency of only applying FD-1 for the Transformer architecture. The Transformer+ UniDrop w/o 2-stage DD means we directly apply conventional data dropout to the sequence instead of our proposed 2-stage strategy. Compared with the full model, its performance also decreases. This shows the necessity of keeping the original sequence for data dropout. To investigate the effects of FD, SD, and DD dropout rates on the UniDrop , we respectively vary them based on the setting (FD= 0 . 1 , SD= 0 . 1 , DD= 0 . 2 ). When varying one dropout component, we keep other dropout rates unchanged. Figure 4 shows the corresponding results. We can observe that the performance of each dropout for Transformer+ UniDrop first increases then decreases when varying the dropout rates from small to large. Especially, varying the rate for FD dropout makes a more significant impact on the model performance since FD contains four feature dropout positions. In contrast, the DD is least sensitive to the dropout rate change, but it still plays a role in the model regularization. Dropout is a popular regularization method for neural networks by randomly dropping some neurons during training (Srivastava et al., 2014). Following the idea, there are abundant subsequent works designing specific dropout for specific architecture, such as StochasticDepth (Huang et al., 2016), DropPath (Larsson et al., 2017), Drop-Block (Ghiasi et al., 2018) for convolutional neural networks, Variational Dropout (Gal and Ghahra-mani, 2016), ZoneOut (Krueger et al., 2017), and Word Embedding Dropout (Gal and Ghahramani, 2016) for recurrent neural networks. Recently, the Transformer architecture achieves great success in a variety of tasks. To improve generalization of Transformer, some recent works propose LayerDrop (Fan et al., 2020a), DropHead (Zhou et al., 2020) and HeadMask (Sun et al., 2020) as structured regularizations, and obtain better performance than standard Transformer. Instead of designing a specific dropout for Transformer, in this work, we focus on integrating the existing dropouts into one UniDrop to further improve generalization of Transformer without any additional cost. Data augmentation aims at creating realistic-looking training data by applying a transformation to a sample, without changing its label (Xie et al., 2020). In NLP tasks, data augmentation often refers to back-translation (Sennrich et al., 2016a), word replacing/inserting/swapping/dropout (Wei and Zou, 2019; Xie et al., 2020), etc. In this work, we adopt simple but effective word dropout as data level dropout in our UniDrop . We, additionally, design a two-stage data dropout strategy. In this paper, we present an integrated dropout approach, UniDrop , to specifically regularize the Transformer architecture. The proposed UniDrop unites three different level dropout techniques from fine-grain to coarse-grain, feature dropout, structure dropout, and data dropout respectively. We provide a theoretical justification that the three dropouts play different roles in regularizing Transformer. Extensive results on neural machine translation and text classification datasets show that our Transformer+ UniDrop outperforms the standard Transformer and various ablation versions. Further analysis also validates the effectiveness of different dropout components and our two-stage data dropout strategy. In conclusion, the UniDrop improves the performance and generalization of the Transformer without additional computational cost and resource requirement. The authors would like to thank the anonymous reviewers for their valuable comments. Xinyu Dai and Lijun Wu are the corresponding authors. This work was partially supported by the NSFC (No. 61976114,61936012) and National Key R&D Program of China (No. 2018YFB1005102).
abstain abstain result abstain objective objective method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result objective method abstain method abstain abstain abstain objective method method abstain objective objective objective objective objective other abstain other other other other other abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain method method abstain method abstain method abstain other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain result other abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other abstain other other method method method abstain method result abstain abstain other other other