Recent studies have determined that the learned token embeddings of large-scale neural language models are degenerated to be anisotropic with a narrow-cone shape. This phenomenon, called the representation degeneration problem, facilitates an increase in the overall similarity between token embeddings that negatively affect the performance of the models. Although the existing methods that address the degeneration problem based on observations of the phenomenon triggered by the problem improves the performance of the text generation, the training dynamics of token embeddings behind the degeneration problem are still not explored. In this study, we analyze the training dynamics of the token embeddings focusing on rare token embedding. We demonstrate that the specific part of the gradient for rare token embeddings is the key cause of the degeneration problem for all tokens during training stage. Based on the analysis, we propose a novel method called, adaptive gradient gating (AGG). AGG addresses the degeneration problem by gating the specific part of the gradient for rare token embeddings. Experimental results from language modeling, word similarity, and machine translation tasks quantitatively and qualitatively verify the effectiveness of AGG. Neural language models have been developed with various architectures during recent years (Graves, 2013; Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017). Despite the improvement in model architectures, models usually share the same process for input and output. They process token embeddings as inputs to compute contextualized features and subsequently project the features into a categorical distribution of tokens at the output softmax layer whose weight is token embedding  Corresponding author. matrix (Merity et al., 2017; Yang et al., 2018; Press and Wolf, 2017). Recent studies have determined that the learned embedding distribution is biased in a common direction, thereby resulting in a narrow cone-shaped anisotropy (Mu and Viswanath, 2018; Ethayarajh, 2019; Gao et al., 2019; Bis et al., 2021). This phenomenon, named the representation degeneration problem by Gao et al. (2019), increases the overall similarity between embeddings, and leads to a problem in which the expressiveness of the token embeddings decreases. Therefore, it is difficult for the model to learn the semantic relationship between the tokens and to generate high quality texts. Existing studies addressing this problem suggest methods that apply post-processing or regularization techniques to all token embeddings based on the observed phenomena owing to the degeneration problem (Mu and Viswanath, 2018; Gao et al., 2019; Wang et al., 2019; Wang et al., 2020; Bis et al., 2021). Although these works improve the quality of token embeddings and generated texts, it is still not clear how token embeddings become degenerate during training procedure. Also, there exists the problem of over regularization for the token embeddings whose semantic relationships are trained well because the above methods are applied for all token embeddings. In this study, we conduct empirical studies about training dynamics of token embeddings, focusing on rare token embeddings. By observing the initial training dynamics of token embeddings grouped based on appearance frequency, we hypothesize that the degeneration of the rare token embeddings triggers the degeneration of the embeddings of the remaining tokens. We show that the entire degeneration problem is mitigated by only freezing rare tokens during training, and we demonstrate that the main cause of the entire degeneration problem is the specific part of the gradient for rare token em-29 beddings. This gradient part pushes away rare token embeddings from the feature vector of the non-rare targets in the current training sample. Based on the analysis, we propose a new method, adaptive gradient gating (AGG). With a dynamic grouping of rare tokens at each training step, AGG solves the entire degeneration problem by gating a specific part of the gradient that is solely about rare tokens. Because AGG is optimized to target the main cause of the degeneration problem, rare token embeddings, it can prevent the over regularization problem about frequent token embeddings which occurs in other methods addressing the degeneration problem. The proposed method is evaluated in three tasks: language modeling, word similarity, and machine translation. The AGG outperforms the baseline and other existing methods in all tasks. In addition, it shows compatibility with other method that addresses the neural text degeneration problem. Via qualitative studies, we identify a correlation between our method and the frequency bias problem of learned embeddings (Gong et al., 2018; Ott et al., 2018). Neural language generative models process text generation tasks as conditional language modeling, in which the model is typically trained by minimizing the negative log likelihood of the training data. With a vocabulary of tokens V = { v 1 , ..., v N } and embedding vectors { w 1 , ..., w N } , where w i corresponds to token v i , at every training step, the model obtains a mini-batch input and target text corpus pair ( x , y ), where x i , y i  V , and y  VT . The conditional probability for the target token y t , P  ( y t | h t ) , where h t is a context feature vector of the t -th position of the generated text conditioned by ( x , y <t ), and  denotes model parameters, which is defined as follows. where w is the output token embedding which roles the weight of the output softmax layer, and I ( y t ) represents the index of token y t . The negative log likelihood loss for an input and target pair ( x , y ), LNLL is expressed as follows. Recent studies on the geometric properties of contextual embedding space have observed that the distribution of embedding vectors is far from isotropic and occupies a relatively narrow cone space(Mu and Viswanath, 2018; Liu et al., 2019; Zhou et al., 2019; Ethayarajh, 2019;). Gao et al. (2019) named this phenomenon the representation degeneration problem . This degeneration problem results in an increase in the overall cosine similarity between token embeddings, making it difficult for the model to learn semantic relationships between tokens. Demeter et al. (2020) demonstrated that the norm information of the token embeddings is so dominant that angle information about the feature vector is ignored when calculating the logits in the output layer. Owing to this structural weakness of the embedding space, embeddings with small norms are always assigned with a low probability, which reduces the diversity of the text generated by the model. Anisotropy of the embedding space is a still problem for the pre-trained large language models, and language models with improved isotropic 30 Methods PPL  I ( W )  Freq Med Rare Total Freq Med Rare Total MLE 16.58 224.24 813.76 20.77 0.426 0.286 0.198 0.293 Freeze 16.48 233.92 3017.53 20.78 0.840 0.651 0.831 0.739 Table 1: Perplexity and I ( W ) for each token groups. embedding space performs well in downstream tasks(Bis et al., 2021; Rajaee and Pilehvar, 2021). Although the problem has been theoretically analyzed in several studies, existing methods are based on the observed phenomena as a result of the problem. To mitigate the phenomena observed from the problem, the post-processing of the embedding vectors(Mu and Viswanath, 2018; Bis et al., 2021) or regularization terms about the phenomena(Gao et al., 2019; Wang et al., 2019; Wang et al., 2020; Zhang et al., 2020) were introduced. These methods are applied to all token embeddings, so there is the problem of over regularization for the embeddings whose semantic relationship is trained well. Also, methodologies based on the training dynamics of the token embeddings concerning the degeneration problem remain subject to study. Frequency bias in embedding space is another problem. Ott et al. (2018) conducted a comprehensive study on the under-estimation of rare tokens in neural machine translation. Gong et al. (2018) observed that embeddings in the language model were biased towards frequency and proposed an adversarial training scheme to address this problem. To analyze the training procedure of token embeddings, we train a Transformer language model at the WikiText-103 dataset from scratch. Whole vocabulary tokens are divided into three groups: frequent, medium, and rare groups. Based on the appearance frequency in the training corpus, the 30%, 50%, and 20% tokens are assigned to the frequent, medium, and rare group. We visualize the initial training dynamics of these groups via the projection of the embeddings into 2D, using singular value decomposition (SVD) projection. As illustrated in Figure 1, rare groups degenerate first, as they emerge from the entire embedding distribution. Subsequently, other groups also start to degenerate, following the degeneration of the rare group. Based on this observation, we hypothesize that the degeneration of rare token embeddings induces the degeneration of non-rare token embeddings . Because Transformer (Vaswani et al., 2017) is representative of the current language models, we adopt the 6-layer Transformer decoder model architecture for an empirical study on the training dynamics of embedding vectors. The model is trained in language modeling task using WikiText-103 dataset (Merity et al., 2018). Experimental details regarding the model and training hyperparameter configurations can be found in the Appendix B. To verify the hypothesis of the previous subsection, we train a model while freezing the rare group token embeddings in their initial states during training, and compare it to the baseline model, where all embeddings are trained with negative log-likelihood loss. In addition, we train the models of various set-31 Methods PPL  I ( W )  Freq Med Rare Total Freq Med Rare Total MLE 16.58 224.24 813.76 20.77 0.426 0.286 0.198 0.293 Freeze tings relative to freezing steps and examine whether the degeneration of rare token embeddings depends on when training of rare embeddings begins. The performance of the models is evaluated in two ways; the likelihood and isotropy of token embeddings. Perplexity (Bengio et al., 2000) is adopted to evaluate the performance of the likelihood of the model. To measure the isotropy of the token embedding distribution, we adopt the partition function Z ( a ) = (cid:80) Ni =1 exp ( w i a T ) defined in Arora et al. (2016), where w i denotes the embedding vector of token v i , and a represents a unit vector. Lemma 2.1. in Arora et al. (2016) demonstrate that if the embedding vectors are isotropic, Z ( a) is approximately constant. Based on this property, we measure the isotropy of an embedding matrix W using I ( W ) , which is defined as follows. where I ( W )  [0 , 1] and X represents the set of eigenvectors of WTW (Mu and Viswanath, 2018; Wang et al., 2020; Bis et al., 2021). Furthermore, we measure the relatedness between the rare and frequent group token embeddings to verify that the degeneration of the frequent group follows the degeneration of the rare group. We calculate the average cosine similarity between the rare and frequent group embeddings to measure the relatedness. Table 1 shows the comparison of the baseline model and the model with frozen rare tokens. We denote the baseline as "MLE" and the freezing method as "Freeze". Surprisingly, the PPL of frequent group tokens and overall I ( W ) improved by simply not training the rare token embeddings. Figure 2 illustrates the change in I ( W ) for the frequent and rare token embeddings, including the similarity between frequent and rare token embeddings at various freezing step settings. Whenever the rare token embeddings start to be trained, their I ( W ) decreases steeply, followed by decreasing I ( W ) of frequent embeddings and increasing similarities between the frequent and rare embeddings. From the analysis in this subsection, we demonstrate that the entire degeneration problem can be solved by solely handling just rare embeddings during the entire training procedure. With T context feature vectors h i ( i  [1 , T ] ) from the training sample, the negative log-likelihood loss gradient for the rare token embedding w r is calculated as follows. where y i denotes the target token for h i , V r is the rare token vocabulary group, and p r | i represents the conditional probability of token v r given h i , which is calculated as [ softmax ( h i WT )] r . We divide the gradient for w r to 3 parts in Eq. 4. Part (a) pulls w r close to the feature vectors whose target tokens are v r . Part (b) pushes away w r from the feature vectors whose target tokens are not rare. Part (c) pushes away w r from the feature vectors whose target tokens are rare. As an extension of the analysis in the previous subsection, we freeze these parts of the gradient with various settings during training to identify the key cause of the degeneration problem. In other words, depending on the settings, the specific gradient parts that will not be used for embedding training is detached from the computation graph during training stage. This can be easily implemented by detach() function of Pytorch (Paszke et al., 2019). All model and training configurations are the same as in the previous sections, except those to be frozen. Table 2 presents the results of the experiments in this subsection. We freeze the parts of the gradient for the rare tokens with three settings. Because part (a) is a key component required to train the token embedding to be aligned to the target, all settings activate part (a). We notice that when part (b) is activated (solely freezing part (c)), I ( W ) decreases and PPL for rare tokens increases almost 10 times compared to when part (b) is frozen. Because activating part (c) is not seen to be negative for PPL and I ( W ) , we conclude that part (b) of Eq. 4 is the bedrock cause for the degeneration problem. From the analysis in this section, we demonstrate that the degeneration problem could be solved to a large extent by mainly addressing the part of the gradient for rare embeddings that pushes away rare token embeddings from non-rare feature vectors . To handle the specific part of the gradient for the rare token embeddings studied in the previous section, we need to properly group the rare tokens. A naive approach can be used to group rare tokens based on the appearance frequency of the training corpus, as described in the previous section. However, this static grouping method is suboptimal because the model is typically trained via mini-batch training. The group of rare tokens that appeared less frequently in recent batch samples is variable in the mini-batch training. Therefore, it is necessary to dynamically group rare tokens based on token appearances in recent batch samples. To consider the token appearances in recent batch samples, we introduce the token counter memory that remembers the number of the appearances of each token during the previous K training steps. For K memories, [ m 1 , ..., m K ], m t  RN represents the number of appearances of each token of N -size vocabulary at the t -th previous training step. Memories are set as zero vectors at the initial stage. At each training step, the token appearance, a  RN , is calculated as the sum of all K memories: a = (cid:80) Kt =1 m t . Based on a , we determine whether token v i is in the rare token group V r as follows. a i K <   v i  V r a i K    v i /  V r , (5) where a i is the i -th component of a , and  is a hyper-parameter in our method that controls the proportion of rare tokens in the entire vocabulary. In this study, we set K to the number of iteration steps during one epoch of training stage. After dynamically grouping the rare tokens at each training step, we need to handle a specific part of the gradient for the rare token embeddings to solve the degeneration problem of all embeddings. To solely control the gradient for rare token embeddings, we introduce a gradient gating method for a parameter x . We define  x as a tensor whose value is the same as x , but detached from the current training graph. This implies that  x is considered a constant, hence, gradient about  x does not exist. In practice,  x can be easily obtained from x using the detach() function of Pytorch (Paszke et al., 2019). With  x , we can gate the gradient for x as follows. where x gated is a new parameter whose value is the same as x , and g  [0 , 1] is a gate tensor. When the x gated is fed to the function f (  ) as input, the gradient for x is gated by g . As we described in section 3, part (b) of Eq. 4 should mainly be handled to solve the degeneration problem. To address part (b) of Eq. 4, given a context feature vector of the i -th position h i , we introduce a gate vector g 1  RN as follows. where g 1 k denotes a k -th component of g 1 . g 1 controls the degree to which rare token embeddings move away from non-rare feature vectors whose targets differ from each rare token embedding. Also, each component of g 1 is calculated based on the rarity of each rare token, a k , so gradient gating for part (b) of Eq. 4 is adaptive for each rare tokens. Although part (c) of Eq. 4, which pushes embeddings away from the feature vectors whose targets are other rare tokens, is not to be seen as the cause of the degeneration problem in section 3, this part also induces the degeneration problem for the certain situation when rare tokens degenerate other rare tokens. To address this, we approximate the multiple levels of rarity in the rare token group to two levels in this paper: less rare' and very rare'. We define the two rarity levels based on the average number of appearances of the entire rare tokens: if the token appearance a k is smaller than the mean of a r where r  V r , corresponding token is a very rare token. For the very rare token embeddings, part (c) of the gradient about embeddings pushes them away from the feature vectors whose targets are less rare tokens that are relatively frequent compared to them. This means that part (c) roles like part (b) in the above situation, which becomes the cause of the degeneration problem. Therefore, we need to handle part (c) of Eq. 4 for very rare tokens. To address part (c) of Eq. 4 for the very rare token embeddings, we introduce another gate vector g 2  RN as follows. where g 2 k is the k -th component of g 2 and  a r is the mean of a r where r  V r . g 2 controls the degree to which very rare token embeddings move away from less rare feature vectors whose targets differ from each very rare token embedding. Also, each component of g 2 is calculated based on the rarity of each very rare token, a k , so gradient gating for part (c) of Eq. 4 is adaptive for each very rare tokens. To calculate the loss of h i , we calculate three logits, z 0 i , z 1 i , and z 2 i , as follows. 1 , 2 . Because our method solely handles the gradient for embeddings, we calculate z 0 i for a gradient about h i , which does not need to be gated. Finally, the negative log-likelihood loss for i -th position L i is computed as follows. L i =  log p 0 I ( y i ) | i  1 ( y i /  V r ) log p 1 I ( y i ) | i  1 ( y i  V r ) log p 2 I ( y i ) | i , (10) where p mI ( y i ) | i = [ softmax ( z mi )] I ( y i ) with m = 0 , 1 , 2 and 1 (  ) denotes the Indicator function. Derivation of the gradient for rare token embeddings,  w r L i , is provided in Appendix A. 5 Experiments We evaluate our method on various tasks including language modeling, word similarity, and machine translation. In the language modeling task, we focus on verifying the diversity of the generated texts. We test the learning of the semantic relationships between tokens on the word similarity task. Finally, we evaluate the quality of generated texts on the machine translation task. For all the experimental results below, we adopt the state-of-the-art model architecture as a baseline to properly demonstrate the effectiveness of our method. Every detail on the experiment, such as model hyper-parameters and training configurations, regard the reproducibility are provided in Appendix B. 34 Method Texts Uniq  Prefix No. 20 Squadron is a Royal Australian Air Force ( RAAF ) support squadron . Setting We conduct experiments using WikiText-103 dataset, which is a significantly large dataset for language modeling task with approximately 103M words and 260K vocabulary size (Merity et al., 2018). Texts in the dataset are preprocessed based on the byte-pair encoding(Sennrich et al., 2016). We adopt the GPT-2 medium architec-ture(Radford et al., 2019), which comprises 24 Transformer decoder layers as a baseline model. Because our method is about learning token embeddings, we train the models from scratch for a maximum of 50k iterations and evaluate them based on the perplexity of the validation set. For hyper-parameter searching, we select   { 0 . 01 , 0 . 02 , 0 . 03 , 0 . 04 , 0 . 05 } for AGG method on the language modeling task. The hyper-parameter sensitivity for the AGG are given in Appendix D. We use three quantitative metrics to evaluate our method: Perplexity, Uniq, and I ( W ) . Related to the likelihood of generated texts, Perplexity quantifies the prediction difficulty over the next token. Uniq (Welleck et al., 2020) quantify the number of unique next-token predictions, measuring the token diversity. As described in section 3, I ( W ) measures the isotropy of the token embedding space. Results We present our results for the testset in Table 3. We denote the baseline method as MLE' and our method as AGG'. We measure Perplexity and Uniq for each token group defined in Section 3. As presented in Table 3, AGG improves the overall metrics for the medium and rare groups while maintaining performance for the frequent token group. This shows that our method not only improves the quality of rare token embeddings, but also the quality of non-rare token embeddings. In particular, for the rare group, the Perplexity score decrease significantly and the number of unique predictions surpasses the human distribution. The I ( W ) for all token embeddings increased over 2 times the baseline. Experimental results of I ( W ) for the embeddings of each frequency groups can be found in Appendix C. Table 5 shows examples of generated texts from MLE baseline and AGG. We also show additional examples of generated texts in Appendix F. Compatibility Neural text degeneration problem is another problem in neural text generative models, where the model generates texts that are less likely to match human word distributions. Existing methods for this problem focus on the diversity of the generated texts by adding an auxiliary loss to the original negative log-likelihood loss (Welleck et al., 2020). Although Welleck et al. (2020) and AGG attempts to address the same problem about diversity, AGG can be compatible with the existing method in the text degeneration problem because AGG does not alter the form of the loss function in MLE training. Table 4 presents the results of the experiments about fusion of unlikelihood train-ing(Welleck et al., 2020) and AGG. We denote the unlikelihood training as UL. From Table 4, we notice that when UL and AGG are fused, it produces a synergistic effect that exceeds the gain of each for the baseline. This indicates that AGG is compatible with methods that address other problems in text generation. Setting We evaluate the semantic relationship between tokens for AGG and the baseline with four word similarity datasets: MEN, WS353, RG65, and RW(Bruni et al., 2014; Agirre et al., 2009; Rubenstein and Goodenough, 1965; Luong et al., 2013). Methods are tested whether the similarity between the given two words in the embedding space is consistent with the ground truth, in terms of Spear-35 Datasets MLE AGG MEN 33.57 55.13 WS353 47.51 56.54 RG65 35.48 65.45 RW 32.13 36.36 Table 6: Performance(Spearman's   100 ) of the models on the four word similarity datasets. man's rank correlation. We adopt cosine distance to compute the similarity between embeddings. We use the same models trained on language modeling tasks with the WikiText-103 dataset for the word similarity task. Results Table 6 presents the result obtained from the evaluation of the word similarity task. From this table, it can be observed that our method outperforms the baseline on overall datasets. Although AGG handles only training of rare tokens, the semantic relationships between all tokens are also well learned. Qualitative studies on semantic alignment between tokens are provided in Appendix E. 5.3 Machine Translation Setting We utilize a dataset from standard WMT 2014 containing 4.5M English  German sentence pairs. The source and target sentences are encoded by 37K shared tokens based on byte-pair encod-ing(Sennrich et al., 2016). We adopt the two version of Transformer(Vaswani et al., 2017) as the baseline model for applying our method: base and big . The model configuration is the same as that proposed in Vaswani et al. (2017). To evaluate the quality of the generated texts, we measure BLEU score (Papineni et al., 2002), which is standard metric for machine translation task. Results Table 7 presents a comparison of our method and other methods in terms of the BLEU score. Our method achieves 1.4 and 1.41 BLEU score improvements on the machine translation task for the base and big baseline models. In addi-Method PPL  Uniq  I ( W )  MLE 15.51 13143 0.377 AGG 15.51 13737 0.813 no g 1 15.48 13018 0.367 no g 2 15.51 13682 0.701 Table 8: Ablation study on gating vector of AGG. tion, our method is better than all other previous works in handling the representation degeneration problem that reported BLEU scores in the same tasks. These results demonstrate the effectiveness of AGG in the quality of the generated texts. While other methods addressing the degeneration problem targets all token embeddings, target of AGG, rare token embeddings, are optimized based on the analysis about the training dynamics of token embeddings. Due to this difference, our method can prevent the over regularization problem for frequent token embeddings, which is the main advantage of AGG compared to other works. Qualitative study about cross-lingual semantic alignment between tokens of the source and target languages is provided in Appendix E. 6 Analysis of AGG 6.1 Ablation Study In our method, AGG, we introduce two gate vectors, g 1 , and g 2 , to handle the gradient for rare and very rare token embeddings. We conduct experiments on these gate vectors. Table 8 presents the results of the ablation studies compared with the MLE and AGG. When g 1 is excluded from AGG (denoted as no g 1 '), Uniq and I ( W ) decreased significantly, because g 1 is the key component for the gradient gating. When g 2 is excluded from AGG (denoted as no g 2 '), Uniq and I ( W ) slightly decrease. Accordingly, we notice that g 2 is important for the gating of gradients fort the very rare token embeddings. Also, we present the analysis about rare token grouping method of AGG. Figure 4 presents the 36 size of the rare token group during initial 1k training steps when the model is trained with WikiText-103 dataset. As presented in the figure, rare group size fluctuate wildly at the initial training stage. We expect for this grouping method to determine an optimal rare token group for the current training step. Table 9 presents the results of ablation study about dynamic grouping. To except dynamic grouping from AGG, we fixed the rare token group after 1 epoch. For this static grouping AGG method, Next-token diversity(Uniq) and the isotropy of the token embedding space( I ( W ) ) perform worse than dynamic grouping AGG. Figure 3 (a) and (b) present the visualizations of the embedding space of baseline MLE and our method. In the figure, applying the AGG method restores the isotropy of the token embedding space. In addition, we observe that the regions occupied by each token group are not disjoint when applying AGG. For baseline, the regions occupied by rare group and the frequent group are disjoint, which is refered as the frequency bias problem of embeddings (Gong et al., 2018). From the analysis of the visualization of the embedding space, we notice that the manipulating the training of the rare token embeddings can alleviate the frequency bias problem. Figure 3 (c) presents the plot of the normalized singular value of embedding matrix for MLE and AGG. Slowly decaying singular values of AGG demonstrate an isotropic distribution of the embedding space. In this study, we analyzed the training dynamics of the token embeddings concerning the representation degeneration problem of the learned embeddings, focusing on the rare tokens. Based on the analysis, we propose an adaptive gradient gating method that solves the problem by solely handling the training for rare token embeddings. Experiments and qualitative studies in various tasks of text generation demonstrate the effectiveness of our method. Beyond the two-level approximation of rarity of rare tokens which is applied to our study, addressing multiple levels of rarity can be an interesting region to study for the future work. This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea gov-ernment(MSIT) [NO.2021-0-01343, Artificial Intelligence Graduate School Program (Seoul National University)], the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Seoul National University in 2022, AIRS Company in Hyundai Motor Company & Kia Corporation through Consortium Fund, and HMC/KIA-SNU AI Consortium SNU-Naver Hyperscale AI Center. A. Graves. 2013. Generating sequences with recurrent neural networks. ArXiv , abs/1308.0850.
abstain abstain abstain method objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method objective objective abstain abstain objective abstain abstain abstain abstain abstain method other other other other other other other other other other other other other other other other other other other method other other method other other objective method abstain abstain method abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain method method method method method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain other method abstain abstain method abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain method abstain method abstain method abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain method objective objective method other other other abstain abstain abstain other