Recently, a lot of research has been carried out to improve the efficiency of Transformer. Among them, the sparse pattern-based method is an important branch of efficient Transformers. However, some existing sparse methods usually use fixed patterns to select words, without considering similarities between words. Other sparse methods use clustering patterns to select words, but the clustering process is separate from the training process of the target task, which causes a decrease in effectiveness. To address these limitations, we design a neural clustering method, which can be seamlessly integrated into the Self-Attention Mechanism in Transformer. The clustering task and the target task are jointly trained and optimized to benefit each other, leading to significant effectiveness improvement. In addition, our method groups the words with strong dependencies into the same cluster and performs the attention mechanism for each cluster independently, which improves the efficiency. We verified our method on machine translation, text classification, natural language inference, and text matching tasks. Experimental results show that our method outperforms two typical sparse attention methods, Reformer and Routing Transformer while having a comparable or even better time and memory efficiency. Transformer (Vaswani et al., 2017) has been widely used and achieved state-of-the-art results in a variety of NLP tasks such as neural machine translation (Bahdanau et al., 2015), text classification, etc. Its good effectiveness benefits from its core component Self-Attention Mechanism which can capture global dependencies well. However, the large calculation and memory cost limit the further application of Transformer on long sequence  The first two authors contributed equally  Corresponding Author tasks due to the complexity of O ( N 2 d ) of Self-Attention. As a result, many research works have been carried out to improve the efficiency of Transformer (Tay et al., 2020b). These efficient Transformers can be roughly divided into two categories: approximation-based (Tay et al., 2020a; Katharopoulos et al., 2020) and sparse pattern-based methods (Qiu et al., 2020; Ho et al., 2019; Beltagy et al., 2020; Liu et al., 2018). Regarding approximation-based methods, some works are based on low-rank approximation (Tay et al., 2020a; Wang et al., 2020) while others are based on kernels (Katharopoulos et al., 2020; Choromanski et al., 2020). Specifically, Lin-former (Wang et al., 2020) adopts a low-rank approximation idea and projects the length dimension of keys and values to a lower dimension ( N  k ). This reduces the complexity to O ( Nkd ) . However, the projection matrix in this method requires that all input sequences must be filled to the same length N , which makes it cannot handle the variable-length sequence well. Linear Transformer (Katharopou-los et al., 2020) uses kernels and the associative property of matrix multiplication to linearize the softmax attention, which reduces the complexity to O ( Nd 2 ) . However, the approximation error to the softmax matrix in Self-Attention can be large in some cases (Xiong et al., 2021). Sparse pattern-based methods introduce sparse patterns into the attention mechanism and limit the number of key vectors that the query vector should pay attention to. Prior work (Child et al., 2019; Qiu et al., 2020; Ho et al., 2019; Beltagy et al., 2020; Liu et al., 2018) proposed to use fixed sparse patterns to improve the efficiency of Self-Attention. For example, Sparse Transformer (Child et al., 2019) restricts the query to focus only on keys that are nearby or at fixed intervals. Such fixed sparse patterns do not consider the similarity between the query and different keys, and directly filter keys according to their location, which re-2390 sults in a degradation of the model effectiveness. More recently, the clustering sparse patterns are proposed. Reformer (Kitaev et al., 2020) and Routing Transformer (Roy et al., 2021) use Locality Sensitive Hashing (LSH) and K-Means algorithms, respectively, to divide the words in the sequence into different clusters, and then the attention operation is restricted within each cluster independently. In this way, they reduce the complexity to O ( N log Nd ) and O ( N  Nd ) , respectively. However, in Reformer and Routing Transformer, both LSH and K-Means only play the role of cluster partitioning, but run separately from the attention network training. In addition, these two methods also have the problem of inconsistency in the similarity measure between the clustering and attention operation. LSH and K-Means respectively use the hash value obtained by random projections and the negative Euclidean distance as the similarity measure between the input vectors while the attention operations use the inner product. Therefore, such a sparse pattern idea often results in the reduced effectiveness. To address the reduced effectiveness issue of many efficient Transformers, especially sparse pattern-based methods, we propose Neural Clustering Method to learn the sparse pattern of the Attention. It can be seamlessly integrated into neural networks for joint training and optimization. In our method, the cluster center (centroid) is updated by a weighted sum of all word hidden states. At the same time, the members of clusters are divided according to the subordinate matrix of the centroids and word hidden states. The optimization of the clustering loss can guide the representation of word hidden states while learning cluster centroids. The integration of the neural clustering method and attention training enables our Neural Clustering Attention to perform better than previous clustering-based sparse attention mechanisms. Our Neural Clustering Method is a general clustering method, in the sense that in addition to being integrated into the network of the specific task, it is can handle clustering tasks alone. Our overall model is called ClusterFormer and it is obtained by replacing the Self-Attention Mechanism in Transformer with Neural Clustering Attention Mechanism. In order to validate the benefits of ClusterFormer, we have carried out comparison experiments of the efficiency and effectiveness respectively. For efficiency, we provide a detailed analysis about the time and memory on dataset 20NEWS of text classification. Results show that our model has a comparable or even better efficiency compared with two typical sparse attention models, Reformer and Routing Transformer. Especially, when the sequence length exceeds 2000, our model, Routing Transfomer and Reformer reduce the memory of Transformer by 53.8%, 60.8% and 31.8% while reducing the training time by 51.4%, 41.8% and 14.4%, respectively on GPU. For effectiveness, we test it on machine translation, text classification, natural language inference, and text matching tasks. Experimental results show that on all tasks, our model consistently outperforms Reformer and Routing Transformer. In particular, our method improves the accuracy by 15.6% and 7.2% on SciTail datasets of natural language inference task compared with Reformer and Routing Transformer, respectively. The major contributions of our work are as follows:  We propose a general end-to-end fuzzy clustering method based on neural network, named Neural Clustering Method, which can dynamically learn weights of each word and then update centroids by weighting all the input words along with the training of specific tasks. We design the Neural Clustering Attention Mechanism based on our proposed clustering method to refactor Self-Attention Mechanism. The experimental results show that our method has comparable efficiency and better effectiveness than typical sparse attention models. The Self-Attention is the core component of Transformer (Vaswani et al., 2017). It extracts sequence features by processing the interaction of three sequence matrices Q , K , and V . Referring to the standard Transformer, its function can be written as follows: Attention ( Q, K, V ) = Softmax ( QKT  d ) V Q, K, V = XWQ , XWK , XWV (1) where X  RN  d model , Q , K, V  RN  d , WQ , WV  R d model  d , N is the length of the se-2391 Word sequence after sort Query after equal chunking Key after equal chunking Word sequence after clustering 1 2 3 4 5 6 7 8 9 1 5 2 3 4 6 7 8 9 1 5 2 3 4 6 7 8 9 1 5 2 3 4 6 7 8 9  1  2  3  4  5  6  7  8  9 X( Word Sequence )  U (Subordinate Matrix ) Clustering Normalization Sort Word Sequence I( Cluster index )    ( Sorted Word Sequence )  ( New Centroids ) C( Centroids )  1  2  3 1 222123 3 3  1  5  2  3  6  4  7  8  9  1  2  3  a  Neural Clustering Method  b  Sorting and Chunking Operation Figure 1: quence, d model is the dimensionality of the model, and d is the dimensionality of the attention head. In Self-Attention Mechanism, the interaction of Q and K gives the N  N attention (weight) matrix, and it leads to the complexity of O ( N 2 d ) , which has been one of the crucial limitation of Transformer. Transformer has been developed into many variants to reduce the complexity of the attention mechanism. In these works, one of the main research directions is to use a sparse attention to substitute the quadratic-cost attention. Some early works (Qiu et al., 2020; Ho et al., 2019; Beltagy et al., 2020) have been proposed to reduce the time complexity by restricting every query to focus only on keys that are nearby or at fixed intervals. This method fixes the sparsity pattern without considering the similarity between queries and keys, limiting its ability to assemble critical information from large contexts. Different from these works, our method attempts to automatically aggregate critical keys for each query based on dependency relationships. Moreover, the clustering-pattern methods were used in Self-Attention to implement a sparse attention. For example, Reformer (Kitaev et al., 2020) and Routing Transformer (Roy et al., 2021) introduce Locality-Sensitive Hashing and K-Means algorithms, respectively, to reduce complexity to O ( N log Nd ) and O ( N  Nd ) . However, in this kind of method, the clustering process and training process are separate, which is a limitation in improving effectiveness. Based on previous researches, we proposed a novel Neural Clustering Method, which can be seamlessly integrated into the network of specific tasks for joint training and optimization to improve effectiveness. In this section, we first introduce our Neural Clustering Method. Then, we introduce our Neural Clustering Attention Mechanism which combines our clustering method and the Self-Attention Mechanism. As shown in Figure 1 (a), our clustering method takes word hidden states X  RN  d and centroid hidden states C  R k  d as inputs. C is initialized randomly in the first layer. Then, we can get the subordinate (similarity) matrix U between word vectors and centroid vectors. It can be defined as: U ij = exp (  ( C i , X j WC )) (cid:80) Nj =1 exp (  ( C i , X j WC )) 1  i  k , 1  j  N (2) where k is the number of clusters and N is the length of sequence. WC  R d model  d model is a parameter matrix. () is a similarity measure function and it is the inner product operation in this scenario. X j is the j -th row of matrix X and C i is the i -th row of matrix C . The subordinate value U ij  [0 , 1] is the normalized similarity value between the i -th centroid vector and emphj-th word vector, and it represents the degree of the word X j belonging to the centroid C i . where i and j represent the index value of the centroid and word, respectively. Then we group the word vectors according to the subordinate matrix U , as follows: I j = Argmax ( U : j ) 1  j  N (4) where U : j is the j -th column of the matrix U and function Argmax () assigns word hidden states to the corresponding cluster according to the maximum subordinate value. Therefore, I  RN represents the cluster index of all the word hidden states. Then, we sort the word vectors according to the cluster indexes I , as follows: XS , I (cid:48) = Sort ( X, I ) (5) where the function Sort () is used to arrange word hidden states belonging to the same cluster to adjacent positions in ascending order of cluster index. XS  RN  d model is the sorted word vectors. I (cid:48)  RN is used to record the original positions of shuffled word hidden states in the sequence and will be used in Eq. 10. Through the above process, we get the grouped and sorted word hidden states XS , as shown in Figure 1 (a). Clustering Loss: Clustering Loss ( L 1 ) is the mean of the negative similarity scores of word hidden states and their belonging centroids, and it will give guidance to learn the optimal clustering scheme. It is defined as follows: L 1 =  1 NN (cid:88) j =1  (cid:16) X j , (cid:98) CI j (cid:17) (6) where X j , (cid:98) CI j represent the j -th word hidden state in the sequence and the updated centroid. The function  () is a similarity measure function and needs to be consistent with Eq. 2. From the above analysis, our Neural Clustering Method is based on the soft clustering. There is a subordinate value between each pair of word vectors and centroid vectors, which can quantitatively describe the fuzzy relationship, so that the clustering can be carried out objectively and accurately. In addition, Neural Clustering Method is based on the neural network, which is easy to integrate into the network corresponding to the target task. The reconstruction of centroid vectors depends on all the word vectors and is based on the continuous optimization for the clustering objective function (as shown in Eq. 6) and the task-specific objective function to get better effectiveness. In addition, we carried out a clustering comparison experiments between our method and traditional clustering methods and observed improvements of our method in effectiveness. See Appendix A for more details. As described in Section 3.1, our Neural Clustering Method groups word vectors with strong dependency into the same cluster and outputs the sorted word vectors XS . Then, we use different matrices to project XS into matrix QS , KS , and VS , as follows: QS , KS , VS = XSWQ , XSWK , XSWV (7) where WQ , WK and WV  R d model  d are weight matrices. QS , KS and VS are matrices Query, Value and Key, respectively. The number of members in each cluster may not be uniform, which makes it difficult for all clusters to perform the attention mechanism in parallel. For parallel computing, after arranging word hidden states in the same cluster to be in adjacent positions, 2393 we chunk them into equal blocks in order, as shown in Figure 1 (b) (essentially similar to the masking of Reformer). The process can be written as follows: QO i = QS (cid:16) ( i  1) (cid:24) N k (cid:25) : i (cid:24) N k (cid:25) (cid:105) KO i = KS (cid:16) ( i  2) (cid:24) N k (cid:25) : i (cid:24) N k (cid:25) (cid:105) 1  i  k (8) where QO i  R w  d and KO i  R 2 w  d are the i -th Query block and Key block respectively. w ( w = Nk ) is the number of members in each block. Matrix VO i has operations similar to KO i . After chunking, Query contains one sequence block while Key and Value consist of two contiguous blocks, which corresponds to L 2 mentioned in Eq. 11. Each token in Query focuses on two blocks of tokens so that the query can cover the words in the same cluster as much as possible. Of course, it does not have to be 2, and can be adjusted. Then, we perform the attention operation within the sequence block in parallel and concatenate the output of each block. where ZO i  R w  d and ZO  RN  d . ZO i is the output of the i -th sequence block after the attention operation. Finally, we recover the shuffled sequence (out-put) to obtain the final result, as follows: Z = Resort ( ZO , I (cid:48) ) (10) where the function Resort () aims to recover shuffled sequence according to the original position record vector I (cid:48) obtained from the Eq. 5. Z  RN  d is the output of Neural Clustering Attention. For the autoregressive modeling, we provide a Masked Neural Clustering Attention Mechanism to prevent the leftward information flow. More details can be found in Appendix B. Centroid Sorting Loss: Centroid Sorting Loss ( L 2 ) is the mean of the negative similarity scores of the adjacent centroid pairs. In Eq. 8, each token in Query block is expected to focus on two continuous blocks of tokens. L 2 makes word hidden states belonging to adjacent clusters are also close to each other. It is defined as follows: L 2 =  1 k (cid:32)(cid:32) k (cid:88) i =2  (cid:16) (cid:98) C i , (cid:98) C i  1 (cid:17)(cid:33) +  (cid:16) (cid:98) C 1 , (cid:98) C k (cid:17)(cid:33) (11) where k is the number of centroids, (cid:98) C i is the i th updated centroid, and the meaning of  () is consistent with Eq. 6. In our method, Clustering Loss, Centroid Sorting Loss, and the loss of target tasks of the model are assigned different weights for joint optimization. More details can be found in Appendix C. 3.3 Analysis of Complexity The complexity of Neural Clustering Attention Mechanism comes from two parts: (i) Neural Clustering Method. In this part, we need to calculate the subordinate matrix between centroid hidden states C  R k  d and the word hidden states X  RN  d , referring to the Eq. 2, which leads to the complexity of O ( Nkd ) . (ii) Attention Mechanism. For this part, we compute attention within the Query block (  R k  w  d ) and Key block (  R k  2 w  d ), referring to the Eq. 9, which leads to the complexity of O ( kw 2 d ) where w = Nk . In summary, the overall complexity is O ( Nkd + kw 2 d ) . When k is set to  N , the complexity is approximately O ( N  Nd ) . In order to verify the effectiveness and efficiency of our method, we carried out the following tasks. We choose Transformer and its clustering-pattern variants (Reformer, Routing transformer) as baseline models. The implementations of the attention layer of Reformer and Routing transformer refer to the open source codes 12 . For a fair comparison, our proposed method and baseline models have the same architecture, except for the attention layer. We validate our model on IWSLT14 German-English and WMT14 English-German benchmarks, which have been widely used for machine translation tasks. For IWSLT14 De-En, it contains about 160K training sentence pairs and is pre-processed by using prepare-iwslt14en2de.sh 3 . For WMT14 En-De, it contains about 4.5 million training sentence pairs and it is pre-processed by using prepare-wmt14en2de.sh 4 . We use the BLEU score as the effectiveness evaluation metric. Some hyperparameters are set: the number of encoder and decoder 1 https://github.com/lucidrains/reformer-pytorch 2 https://github.com/lucidrains/routing-transformer 3 https://github.com/pytorch/fairseq/blob/master/examples /translation/prepare-iwslt14.sh 4 https://github.com/pytorch/fairseq/blob/master/examples /translation/prepare-wmt14en2de.sh 2394 Model IWSLT14 De-En WMT14 En-De Transformer  (Vaswani et al., 2017) 34.4 27.3 / 26.4 Reformer  (Kitaev et al., 2020) 34.0 26.3 / 25.4 Routing Transformer  (Roy et al., 2021) 32.5 24.3 / 23.6 ClusterFormer 34.9 27.4 / 26.5 Table 1: Test BLEU on IWSLT14 (De-En) and WMT14(En-De). layers L = 6, the number of centroids k = 3. The dimension of word embedding and model d model = 512. Specifically, for IWSLT14, the number of heads is set to 4 and d ff = 1024. For WMT14, the number of heads is set to 8 and d ff = 2048. As shown in Table 1, our method boosts effectiveness on both datasets. Specifically, the Tokenized BLEU score is improved by at least 1.5% compared with other models on IWSLT14 datasets. Compared with the latest models Reformer and Routing Transformer, ClusterFormer respectively has 2.6% and 7.4% improvement. Our method shows the same trend on WMT14 datasets. Especially, compared with Reformer and Routing Transformer, the Tokenized BLEU score of ClusterFormer respectively has 4.2% and 12.8% improvement and the sacreBLEU score respectively has 4.3% and 12.3% improvement. We validate our model on five text classification tasks. CR (Hu and Liu, 2004): Customer reviews composed of positive or negative product reviews; MR (Pang and Lee, 2004): Movie reviews divided into positive and negative categories; SUBJ: Subjectivity dataset where the target is to classify a text as being subjective or objective; MPQA (Wiebe et al., 2005): Opinion polarity detection subtask. 20NEWS: A international standard dataset for text classification, text mining, and information retrieval research. The dataset collects about 20,000 newsgroup documents, divided into a collection of newsgroups on 20 different topics. Accuracy is used as the evaluation metric for these datasets. In addition, for all datasets, word embeddings are initialized by GloVe (Pennington et al., 2014) with 300-dimension. Some hyperparameters are set: The number of encoder layers L = 2, the dimension of model d = 300, the number of heads h = 4, and the number of centroids k is adjusted near the square root of the max length. As shown in Table 2, ClusterFormer outperforms all baseline models and improves the test accuracy by at least 3.16%, 1.70% for CR and SUBJ datasets, respectively. In addition, on the MPQA dataset, ClusterFormer achieves a comparable result with MPSAN. We also carry out the text classification task on the long text dataset 20NEWS. The accuracy for the 20NEWS dataset increases at least 0.24% compared with other models. In addition, compared with the latest models Reformer and Routing Transformer, our model respectively has 6.1%, 3.8%, 1.6%, 2.0%, 2.6% and 10.0%, 4.9%, 2.0%, 11.3%, 3.1% improvement for CR, MR, SUBJ, MPQA and 20NEWS datasets. In this section, we conduct Natural Language Inference tasks on SNIL, SciTail datasets, and Text Matching tasks on Quora, WikiQA datasets. SNLI (Bowman et al., 2015) is a benchmark dataset 2395 Model SNLI SciTail Quora WikiQA map mrr DELTA (Han et al., 2019) 80.7     Bigram-CNN (Yu et al., 2014)    0.619 0.628 Transformer  (Vaswani et al., 2017) 83.7 76.6 85.4 0.601 0.613 Reformer  (Kitaev et al., 2020) 78.6 67.3 74.3 0.587 0.603 Routing Transformer  (Roy et al., 2021) 76.3 72.6 81.5 0.560 0.574 ClusterFormer 83.9 77.8 85.4 0.630 0.648 Table 3: Experimental results on Neural Language Inference (NLI) and Text Matching tasks. for natural language inference. There are 570k human-annotated sentence pairs with four labels. SciTail (Khot et al., 2018) is an entailment classification dataset constructed from science questions and answers. Quora Question Pairs is a dataset for paraphrase identification with two classes indicating whether one question is a paraphrase of the other. The evaluation metric for these three data sets is Accuracy. WikiQA (Yang et al., 2015) is a retrieval-based question answering dataset based on Wikipedia, which is composed of 20.4k/2.7k/6.2k (train/dev/test) samples. The mean average precision (MAP) and mean reciprocal rank (MRR) are used as the evaluation metrics. For SNIL and Quora datasets, word embeddings are initialized by GloVe (Pennington et al., 2014) with 300 dimensions. For the rest, we use random word embedding vectors with 300 dimensions. Some hyperparameters are set: L = 1, the number of heads h = 6 and the number of centroids k = 3. As shown in Table 3, our model achieves the best results for most datasets. Specifically, the accuracy of our model is at least 1.6% higher than baseline models on the SciTail dataset. On WikiQA, our model improves the result by at least 1.8% and 3.2% in MAP and MRR evaluation metrics, respectively. Our model and Transformer have considerable effectiveness on SNLI and Quora datasets. In addition, compared with the latest models Reformer and Routing Transformer, our model has 6.7%, 15.6%, 14.9%, and 10.0%, 7.2%, 4.8% improvement for SNLI, SciTail, Quora datasets. For the WikiQA dataset, the score increases 7.3% and 7.5% by our model in MAP and MRR compared to Reformer. The score increases 12.5% and 13.0% compared to Routing Transformer. In this section, we test the effect of different clustering numbers ( k ) on the effectiveness and efficiency. We test our model on the 20NEWS dataset of text classification tasks with a NVIDIA V100 (16GB) GPU. Some hyperparameters are set: the number of encoder layers L is 2, the dimension of the model d is 300, the batch size is 64, and the max sequence length N is 1500. From Table 4, we can draw the following conclusions: (i) Accuracy of our model: In general, within a certain range during the growth of k , the performance of our model is relatively stable. When the value of k goes beyond a certain threshold, the performance of our model degrades; (ii) Memory cost of our model: As the number of centroids k increases, the memory cost of the model decreases first and then increases; (iii) Training time of our model: As the number of centroids k increases, the training time of the model also decreases first and then increases. Therefore, according to this law, our method can simultaneously gain both the effectiveness and efficiency of the model by determining an appropriate k value through finite experiments. In this section, we provide an ablation experiment about the two kinds of clustering losses. We verify the effectiveness of the two loss modules by assigning different weight. Some hyperparameters 2396 are set: the number of encoder layers L is 1, the dimension of model d is 300, the batch size is 128 and the max sequence length N is 500. From Table 5, the experimental result shows that both L 1 and L 2 contribute to the performance. For example, on dataset SciTail, the accuracy with the best result is improved by 1.46% (acc) compared with the result without the two losses. On dataset WikiQA, the accuracy with the best result is improved by 3.10% (map), 1.89% (mrr) compared with the result without the two losses. In this section, we provide a comparison experiment on dataset 20NEWS about the time and memory cost for different models. About the dataset, its average sequence length is approximately 280 and the maximum sequence length exceeds 10,000. To compare time and memory cost, we set the range of sequence length N as (0, 2000] and batch size to 20. We test the memory and time cost on a NVIDIA V100 GPU. We take the time of 1000 steps forward propagation of the model as the inference time, and the time of 1000 steps forward and back propagation as the training time. As shown in Figure 4, as the sentence length increases, both Routing Transformer and our model can significantly reduce memory cost compared to Transformer. When N exceeds 2000, our model, Routing Transfomer and Reformer reduce the memory by 53.8%, 60.8%, and 31.8%, respectively. TransFigure 4: Memory cost versus the sequence length. former increases significantly with increasing sequence length, while our model and Routing Transformer have a relatively small increase on GPU devices. When N is 2000, our model, Routing Transfomer and Reformer reduce the training time by 51.4%, 41.8%, and 14.4%. However, the inference speed of these improvements is inferior compared with Transformer, which may be caused by the decrease of the model parallelism. The above analysis fully demonstrates the efficiency and effectiveness of our proposed Neural Clustering Mechanism. In this paper, we propose a Neural Clustering Attention Mechanism to address the reduced effectiveness issue in sparse attention methods. This issue is mainly caused by the introduction of a sparse pattern that is separated from the target task or does not consider the similarity between words. In our method, we design a neural clustering algorithm to better capture critical pairs of dependencies. We integrate this clustering algorithm and the neural network to jointly train and optimize with specific tasks together to further contribute to the effectiveness and efficiency. The experimental results show that our model can achieve better effectiveness and a comparable or even better efficiency, compared with the latest typical sparse attention models, Reformer and Routing Transformer. This work is supported in part by the state key development program of China (grant No.2017YFE0111900), Natural Science Foundation of China (grant No.61772363), PolyU internal fund (grant No.1-BD47) under the research project (P0039657) of the Hong Kong Polytechnic University, and the Beijing Academy of Artificial Intelli-gence(BAAI).
abstain abstain abstain abstain method abstain result result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain method abstain abstain method abstain result abstain method result result abstain result result objective objective result other other other other other other other other other abstain other other other objective abstain method abstain method abstain method abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain method abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain objective abstain method objective result other