Neural networks equipped with self-attention have parallelizable computation, light-weight structure, and the ability to capture both long-range and local dependencies. Further, their expressive power and performance can be boosted by using a vector to measure pairwise dependency, but this requires to expand the alignment matrix to a tensor, which results in memory and computation bottlenecks. In this paper, we propose a novel attention mechanism called Multi-mask Tensorized Self-Attention (MTSA), which is as fast and as memory-efficient as a CNN, but significantly outperforms previous CNN-/RNN-/attention-based models. MTSA 1) captures both pairwise (token2token) and global (source2token) dependencies by a novel compatibility function composed of dot-product and additive attentions, 2) uses a tensor to represent the feature-wise alignment scores for better expressive power but only requires parallelizable matrix multiplications, and 3) combines multi-head with multi-dimensional attentions, and applies a distinct positional mask to each head (subspace), so the memory and computation can be distributed to multiple heads, each with sequential information encoded independently. The experiments show that a CNN/RNN-free model based on MTSA achieves state-of-the-art or competitive performance on nine NLP benchmarks with compelling memoryand time-efficiency. Recurrent neural network (RNN) and convolutional neural network (CNN) have been broadly used as context fusion modules for natural language processing (NLP) tasks. Recently, RNN/CNN in conjunction with an attention mechanism has been proven to be effective for contextual feature modeling in a wide range of NLP tasks, including sentiment classification (Li et al., 2018), machine translation (Bahdanau et al., 2015), reading comprehension (Seo et al., 2017; Yu et al., 2018), etc. More recently, self-attention mechanisms have been developed for context fusion and syntactic dependency modeling with the advantage of fewer parameters, more parallelizable computation, and better empirical performance (Hu et al., 2017; Vaswani et al., 2017; Shen et al., 2018a). In addition, neural networks based solely on self-attention mechanisms have achieved state-of-the-art quality on many NLP tasks, e.g., machine translation (Vaswani et al., 2017), sentence embedding (Shen et al., 2018a) and semantic role labeling (Tan et al., 2017). Self-attention mechanisms can be categorized into two classes according to the type of dependency each aims to model. The first category is token2token self-attention (Hu et al., 2017; Vaswani et al., 2017; Shen et al., 2018a) that captures syntactic dependency between every two tokens in a sequence. An efficient dot-product compatibility function is usually deployed to measure this pairwise dependency (Vaswani et al., 2017). In contrast, additive compatibility function captures the dependency by multi-layer perceptron (MLP), and can usually achieve better performance (Britz et al., 2017). Its expressive power can be further improved if expanded to multiple dimensions (Shen et al., 2018a). This multi-dim self-attention empirically surpasses dot-product one, but suffers from expensive computation and memory, which grow linearly with the number of features and quadratically with the sequence length. Hence, it is not scalable to long sequences in practice. The second category is source2token self-attention (Liu et al., 2016; Lin et al., 2017; Shen et al., 2018a) aiming to capture global dependency, i.e., the importance of each token to the entire sequence for a specific task. Its time and space complexities grow linearly, rather than quadratically, with the sequence length. Hence, it is empirically efficient in terms of memory and computation even if expanded to multiple dimensions, i.e., using a vector of feature-wise scores instead of a scalar for the global dependency. But, it is hard to reach state-of-the-art performance on NLP tasks due to the lack of pairwise and local dependencies. In this paper, we propose a novel attention mechanism called multi-mask tensorized self-attention (MTSA), for context fusion. In MTSA, 1) the pairwise dependency is captured by an efficient dot-product based token2token self-attention, while the global dependency is modeled by a feature-wise multi-dim source2token self-attention, so they can work jointly to encode rich contextual features; 2) self-attention alignment scores are tensorized for more expressive power in that each pair of tokens has one score for each feature, but no tensor computation is required other than simple and efficient matrix multiplications when implemented; 3) the tensors above are computed in multiple subspaces (i.e., in a multi-head fashion) rather than in the original input space, so the required memory and computation can be distributed to multiple subspaces; and 4) a distinct positional mask is applied to each head in order to encode rich structural information such as the sequential order and relative position of tokens. In the experiments, we build CNN/RNN-free neural networks based on MTSA for sentence embedding and sequence tagging tasks, including natural language inference, semantic role labeling, sentiment analysis, question-type classification, machine translation, etc. The results demonstrate that MTSA achieves state-of-the-art or competitive performance on nine benchmark datasets. To summarize the comparison of MTSA with recently popular models, we show the memory consumption and time cost vs. sequence length respectively in Figure 1(a) and 1(b) on synthetic data (batch size of 64 and feature channels of 300). On the SNLI (Bowman et al., 2015), a public dataset for language inference, as shown in Figure 1(c), MTSA achieves the best result but is as fast and as memory-efficient as the CNNs (all baselines and the benchmark are detailed in Section 4). Notations: 1) lowercase denotes a vector; 2) bold lowercase denotes a sequence of vectors (stored as a matrix); and 3) uppercase denotes a matrix or tensor. Given an input sequence of token embeddings or memory slots x = [ x 1 , . . . , x n ]  R d e  n , and a vector representation of a query q  R d q , attention mechanism (Bahdanau et al., 2015; Lu-ong et al., 2015) computes an alignment score between each token x i and q by a compatibility function f ( x i , q ) , which aims to measure the depen-dency/relevance between x i and q , or the attention of q to x i , w.r.t. a given task. The scores are transformed into probabilities through a softmax function. These probabilities are then used as weights to sum all the tokens and generate a contextual embedding for q , i.e., p ( z | x , q ) = softmax( a ) , a = [ f ( x i , q )] ni =1 , s = n (cid:88) i =1 p ( z = i | x , q )  x i = E i  p ( z | x ,q ) [ x i ] , (1) where a  R n denotes the vector of n alignment scores, p ( z | x , q ) is the categorical distribution for attention probabilities, which is derived from applying softmax function to a . And, s  R d e is the output vector for the query q . There are two major types of compatibility functions, leading to the two most frequently used attention mechanisms. The first one is dot-product or multiplicative compatibility function (Eq.(2)), which composes dot-product attention mechanism (Luong et al., 2015) using cosine similarity to model the dependencies. The other one is additive or multi-layer perceptron (MLP) compatibility function (Eq.(3)) that results in additive attention mechanism (Bahdanau et al., 2015) using MLP to model the dependencies. where W ( d 1)  R d i  d e , W ( d 2)  R d i  d q , W ( a )  R d a  ( d e + d q ) , w  R d a are learnable parameters, (cid:104) , (cid:105) denotes inner-product. Empirically, networks with additive attention usually outperform those with dot-product attention, but require more computation time and memory (Britz et al., 2017). Multi-dim attention mechanism (Shen et al., 2018a) expands the alignment score in previous attention mechanisms to a vector for feature-wise scores, each computed on a feature dimension. It has greater capacity to model complex dependencies, and can handle context variation and polysemy problems harassing many NLP tasks. In particular, it replaces vector w T  R 1  d a in additive compatibility function (Eq.(3)) with a matrix W  R d e  d a , and thus produces d e scores to describe the attention of q to x i . Self-attention mechanism is a special case of attention mechanisms, where the query q stems from the input sequence itself. Self-attention mechanisms can be classified into token2token or source2token self-attention mechanism according to the type of dependency each aims to model. A) Token2token self-attention mechanism (Vaswani et al., 2017; Shen et al., 2018a) aims at producing a context-aware representation for each token in light of its syntactic dependencies on other tokens from the same sequence. Two examples of token2token self-attention are 1) scaled dot-product self-attention which composes the multi-head self-attention (Vaswani et al., 2017), and 2) masked self-attention used in directional self-attention (Shen et al., 2018a). A.1 ) Scaled dot-product attention mechanism (Vaswani et al., 2017) in general form has three arguments: query tokens q  R d i  m , key tokens k  R d i  n and value tokens v  R d h  n associated with the key tokens. It uses a scaled dot-product function to model the relationship between each query and key, and finally outputs a sequence s = [ s 1 , . . . , s m ]  R d h  m such that s =sdpAttn( q , k , v ) (cid:44) v softmax( q T k (cid:112) d q ) T (4) A special case of this mechanism is that the three input arguments are derived from the same source, i.e., q / k / v = f q / k / v ( x ) , which can be referred to as a token2token self-attention, namely scaled dot-product self-attention. As for multi-head attention mechanism, the input is projected into multiple subspaces, then parameter-untied scaled dot-product attention is applied to the embeddings in each subspace. The results for multiple subspaces are concatenated to form the final output s , i.e., s = W ( o ) [ H 1 ; . . . ; H h ] , (5) where H c = sdpAttn( W qc q , W kc k , W vc v ) . A.2) Masked self-attention mechanism (Shen et al., 2018a) uses multi-dim compatibility function to model the dependency between every two tokens in a sequence, and uses positional mask to encode sequential information. It overcomes inherent problem appearing in self-attention compared to RNNs on the lack of sequential information. Its compatibility function is defined as f ( x i , x j )= c  tanh { ( W ( m ) [ x i ; x j ]+ b ( m ) ) /c } + M i,j (6) where c is a constant scalar, W ( m )  R d e  2 d e is learnable weight matrix, and M is a positional mask with each entry M i,j  { , 0 } . When M i,j =  , applying softmax function to the alignment scores results in a zero attention probability, which cuts off the attention of x j to x i . Hence, masked self-attention with an asymmetric mask, where M ij (cid:54) = M ji , can encode sequential or other structural information (Shen et al., 2018a; Im and Cho, 2017). To this end, two positional masks have been proposed to encode the forward and backward order information respectively, i.e., M fwi,j = (cid:26) 0 , i < j  , otherwise M bwi,j = (cid:26) 0 , i > j  , otherwise Furthermore, directional self-attention (DiSA) (Shen et al., 2018a) concatenates the features produced by masked self-attention mechanisms with the forward and backward positional masks (i.e., M fw , M bw ), leading to context-ware representations with bi-directional information encoded. B) Source2token self-attention mechanism (Liu et al., 2016; Lin et al., 2017; Shen et al., 2018a) is designed for sentence embedding or sequence compression, which is based on the importance of each token x i to the entire source sequence x for a specific task. Specifically, it removes the query q from the compatibility function f ( x i , q ) when computing the alignment score. For example, the compatibility function of additive source2token self-attention mechanism is to simply remove q from Eq.(3). In this section, we firstly elaborate on tensorized self-attention (TSA) in Section 3.1, which captures both pairwise and global dependencies by combining the two types of self-attention mechanisms introduced in Section 2.2. Then, we extend TSA to multi-mask tensorized self-attention (MTSA) in Section 3.2 by applying different positional masks to TSA in multiple subspaces (multi-head fashion). Lastly, in Section 3.3, we present an efficient computation scheme for MTSA without any high-rank tensor computation involved even if tensorized alignment scores are used. Tensorized self-attention (TSA), whose structure is illustrated in Figure 2, is a neural mechanism that can be trained to model both pairwise and global dependencies, while any previous self-attention mechanism only focuses on one type of dependencies. TSA models both types by combining the aforementioned token2token and source2token self-attention mechanisms. This generates an n  n  d h tensor containing the alignment scores between every two tokens on each feature dimension. These scores are then normalized and transformed into probability weights, which are used to sum all dependent tokens and then generate the contextual embedding for each input token. We will demonstrate later in Section 3.3 that only matrix rather than tensor operation is required when executing the procedures above. To facilitate the elaboration of proposed models and keep the consistent notation with prior attention mechanisms, TSA first projects the input embeddings x into three spaces to represent the query, key and value tokens, respectively. q = W ( t 1) x , k = W ( t 2) x , and v = W ( t 3) x , (7) where W ( t 1) , W ( t 2)  R d i  d e and W ( t 3)  R d h  d e are learnable weights for projections. TSA then integrates two kinds of compatibility functions from two self-attention mechanisms respectively. Firstly, the scaled dot-product self-attention is used to capture dependency between every two tokens. Dot-product operations are fast, and sufficient to model the pairwise dependency in most tasks. Its compatibility function is f t ( k i , q j ) = (cid:104) k i , q j (cid:105) / (cid:112) d i ,  i, j  [ n ] , (8) where (cid:104) , (cid:105) is inner-product operation. Then, a multi-dim source2token self-attention mechanism is used to estimate the contribution of each token to the given task on each feature dimension. It aims at capturing the importance of each token to the entire input sequence w.r.t. the task, i.e., the global dependency . The multi-dim extension only linearly increases the memory and computation of source2token self-attention by a multiplicative factor d h , but is essentially helpful to improve expressive capability in line with prior works (Shen et al., 2018a). Its compatibility function is f s ( k i ) = W ( s 2)  m ( W ( s 1) k i + b ( s 1) )+ b ( s 2) , (9) where  i  [ n ] , W ( s 1)  R d a  d i , W ( s 2)  R d h  d a are the learnable weights, and  m (  ) is an activation function. The compatibility function used in TSA broadcasts the scalar alignment score f t ( k i , q j )  R computed by the token2token self-attention to all d h feature dimensions, and then adds them to the feature-wise score vector f s ( k i )  R d h computed by the source2token self-attention. In addition, the positional masks from masked self-attention (in Section 2.2) are also integrated to encode sequential and structural information. These yield following compatibility function of TSA. For each query token q j , a softmax function is applied to the alignment scores [ f tsa ( k i , q j )] ni =1 on each feature dimension, resulting in a categorical distribution over all value tokens [ v i ] ni =1 based on corresponding key tokens [ k i ] ni =1 . The probability of token q j attending to v i on the l th feature dimension (i.e., z l = i ) is p ( z l = i | k ,q j ) (cid:44) [ p ji ] l (cid:44) e [ f tsa ( k i ,q j ) ] l (cid:80) ng =1 e [ f tsa ( k g ,q j )] l , (11) where,  i, j  [ n ] ,  l  [ d h ] . TSA outputs a contextual embedding for each input token on every feature dimension as the weighted sum of all the value token embeddings on that dimension, where the weights are provided by the probabilities in Eq.(11). It is the expectation of sampling a value token embeddings on each feature dimension according to the feature-wise probability distribution, i.e., s (cid:44) [ s j ] nj =1 , where (12) s j (cid:44) (cid:104) E i  p ( z l | k ,q j ) ([ v i ] l ) (cid:105) d h l =1 = (cid:88) n i =1 p ji  v i 3.2 Multi-Mask Tensorized Self-Attention (MTSA) Mechanism Rather than computing attention in the original input space, multi-head attention (Vaswani et al., 2017) projects the input sequence to multiple subspaces, applies attention to the projected embedding in each subspace, and concatenates their outputs at last. The computations associated with multiple heads can be completed in parallel. By using adequate heads, each with a low-dimensional subspace (i.e., the representation dimension for each head is updated by d h  d h /h where h is the number of head), it reduces parameters and memory/computation cost and increases diversity of the attention. In addition, to encode different kinds of sequential or structural information, multiple different positional masks (e.g., forward, backward and multi-length window) can be further applied to the multiple heads. The memory-/time-efficiency and expressive power of TSA can be improved by using the combination of the multi-head and multi-mask techniques introduced above. By writing TSA mechanism as a function TSA( x , M ) with input sequence x  R d e  n and a positional mask M  R n  n , and the output given by Eq.(12), multi-mask tensorized self-attention (MTSA) produces s = W ( o ) [ H 1 ; . . . ; H h ] , (13) where H c = TSA c ( x , M c ) , where W ( o )  R h  d h  h  d h , h is the number of heads, TSA c denotes the c th parameter-independent TSA block that produces a d h -dim representation in the c th subspace, M c represents the positional mask applied to attention in the c th subspace, [  ; . . . ;  ] denotes a vertical concatenation operation, and s  R h  d h  n is the output of MTSA. In our experiments, we apply forward mask to half of the heads and apply backward mask to the other half to encode bi-directional order information of the input sequence. As shown in Eq.(10) and Eq.(11), TSA or each head of MTSA needs to compute the attention scores and probabilities as n  n  d h tensors. In accordance with multi-dim self-attention (Shen et al., 2018a), this makes TSA more expressively powerful and improves the final performance for sequence modeling, but terribly leads to memory explosion and computational bottleneck on long sequences with large n and d h . Fortunately, in MTSA, it is possible to significantly reduce the demand on computations to matrix-only operations by exploring the computational structure. A memory-optimized and highly-parallelizable computation scheme for MTSA is given in Algorithm 1. For each head, the score matrices of token2token and source2token are computed in steps 3 and 4 respectively. Then, we combine token2token scores with the positional mask to form a new mask in step 5, and compute the d h  n output embedding with the weighs from the multi-dim source2token self-attention in step 6. Finally, in step 7, we apply the new mask from step 5 to the weighted embedding from step 6 and complete the normalization. This procedure generates the exactly same output as Eq.(13)but no any tensor operation is incurred. We compare MTSA with commonly-used context fusion baselines on several NLP tasks 1 . When addressing a sentence embedding problem, a multi-dim source2token self-attention is applied on the top of context fusion module to produce the sequence embedding. Codes are implemented in Python with Tensorflow and executed on a single NVIDIA GTX 1080Ti graphics card. In addition, data for both time cost and memory consumption are collected under Tensorflow-1.7 with CUDA9 and cuDNN7. The context fusion baselines include 1) BiLSTM (Graves et al., 2013): 600D bi-directional LSTM consisting of 300D forward plus 300D backward LSTMs, 2) Bi-GRU (Chung et al., 2014): 600D bi-directional GRU, 3) Multi-CNN (Kim, 2014): three CNNs with 200D kernels to model 3/4/5-grams respectively, 4) Hrchy-CNN (Gehring et al., 2017): 3-layer 300D stacked CNN with kernel size 5, gated linear units (Dauphin et al., 2016) and residual connections (He et al., 2016), 5) Multi-head (Vaswani et al., 2017): 600D multi-head self-attention with 8 heads (75-1 Codes for Experiments are released at https:// github.com/taoshen58/mtsa . dim subspace per head) and positional embedding used by Vaswani et al. (2017), 6) DiSA (Shen et al., 2018a): 600D directional self-attention mechanism consisting of 300D forward and 300D backward masked self-attentions, and 7) Bi-BloSA (Shen et al., 2018c): 600D bidirectional block self-attention with intra-/inter-block self-attention, aiming to reduce the time and space complexities of multi-dim self-attention by using hierarchical structure. Natural language inference (NLI) aims at speculating on the relationship between a premise and a corresponding hypothesis, where the relationship could be entailment , neutral or contradiction . In experiments, we first compare MTSA with other baselines on the Stanford Natural Language Inference (Bowman et al., 2015) (SNLI) dataset. Following the method of applying sentence-encoding model to NLI given by Bowman et al. (2016), two parameter-tied sentence-encoding models are used to generate embeddings for premise and hypothesis, resulting in s p and s h respectively. The concatenation of s p , s h , s p  s h and s p (cid:12) s h representing the relationship is passed into a 3-way neural classifier for final prediction. The experimental results of the models from the official leaderboard, baselines, and MTSA are shown in Table 1. MTSA achieves state-of-the-art performance with less time and memory cost. Compared to the methods from the leaderboard, MTSA outperforms RNN-based encoders (e.g., Residual stacked enc.), RNN+attention encoders (e.g., Deep Gated Attn.) and even parsing trees based encoders (e.g., Gumbel TreeLSTM enc.) by a large margin. Compared to the two competitive self-attention networks with complicated and expensive training computations, MTSA trained in end-to-end manner achieves the same state-of-the-art performance by using much fewer parameters and less computational time. Compared to baselines, MTSA is 4  5  faster than RNN-based models and outperforms CNN-based models given a similar number of parameters and computation time. Moreover, compared to the dot-product self-attention (Multi-head), MTSA costs similar time and memory but performs more expressively powerful self-attention, and thus achieves better performance. Furthermore, compared to the multi-dim self-Model |  | Time/Epoch Inf. attention (DiSA and Bi-BloSA), MTSA uses much less memory and time but even produces much better prediction quality. In addition, to further improve the state-of-the-art performance, in contrast to training from scratch, a language model built on the Transformer (Vaswani et al., 2017) unsupervisedly pretrained on large English corpus (detailed by Radford et al. (2018)) is transfered for the baseline and proposed models for sentence-encoding based NLI tasks. As shown in Table 2, MTSA integrated with pretrained language model can achieve new state-of-the-art accuracy on both SNLI and Multi-Genre Natural Language Inference (MultiNLI) (Williams et al., 2017) 2 among all sentence-encoding mod-2 All test results are Evaluated on Kaggle official Model |  | Inf. An ablation study of MTSA is shown in Table 3 to verify the capability of its each part in context fusion. The results show that token2token (model-ing pairwise dependency), source2token (model-ing global dependency), and positional masks (en-coding sequential information) all contribute important information to sequence modeling, and the contributions are complementary. To verify the capability of MTSA in generating context-aware representation of each token, we compare it with baselines on semantic role labeling (SRL) task, which aims to tag each token from an input sequence with a label for its semantic role. Particularly, given a sentence, the goal of SRL is to identify the arguments of each target verb into semantic roles, which can benefit many downstream NLP tasks. SRL has two steps: websites: https://www.kaggle.com/c/multinli-matchedopen-evaluation and https://www.kaggle.com/c/multinlimismatched-open-evaluation Models Training Development WSJ Test Brown Test Time P R F1 Comp. We follow the experimental setup in Tan et al. (2017), where the SRL task is treated as a BIO tagging problem. Tan et al. (2017) designed a deep attentive neural net by stacking multi-head self-attention, named as deepatt, to perform context fusion, whose output is then passed to a neural classifier to make the final decision. The results achieved by previous methods, baselines, and MTSA are shown in Table 4, which demonstrates that MTSA achieves new state-of-the-art performance on the CoNLL-05 dataset by costing similar training time as CNN and multi-head self-attention baselines. We evaluate the models on five sentence classification benchmarks for different NLP tasks, which include 1) CR (Hu and Liu, 2004): customer reviews of various products to predict whether the review is positive or negative, 2) MPQA (Wiebe et al., 2005): an opinion polarity detection subtask of the MPQA dataset, 3) SUBJ (Pang and Lee, 2004): subjectivity dataset where a label indicates whether a sentence is subjective or objective, 4) TREC (Li and Roth, 2002): question-type classification dataset which classifies the question sentences into six classes, 5) SST-5 (Socher et al., 2013): the Stanford Sentiment Treebank dataset with five sentiment labels. The reported accuracies for CR, MPQA, and SUBJ are the mean of 10-fold cross validation. The accuracies for TREC are the mean of five runs on the dev set, and the accuracies for SST-5 are the mean of five runs on the test set. All standard deviations are shown in parentheses. The prediction accuracies achieved on these five benchmarks are shown in Table 5. MTSA achieves the best prediction accuracy on CR, MPQA, TREC and SST-5 benchmarks with better time efficiency and a lower memory load. We also evaluate proposed model on WMT 2014 English-German translation task for exhaustive comparisons with multi-head attention. We replace multi-head self-attention modules in the encoder of official Transformer implementation with MTSA module and do not tune the hyperparame-ters. Although our computation resources is limited, we use two training setups and also introduce t -test to ensure that MTSA consistently outperforms multi-head self-attention in Transformer. For Setup1 , we use default hyperparameter set of transformer base single gpu provided by official implementation with 1  P100 , batch size of 2048 and training step of 250K, and report BLEU value for the last checkpoint. For Setup2 , we use the hyperparameter set of transformer base with the modification of 1) using 4  instead of 8  P100, 2) increasing batch size from 4096 to 6144 per GPU, and 3) using training step of 133K. As shown in Table 6, with small p-value for both training setup 1 and 2, the encoder with MTSA significantly outperforms that with multi-head self-attention, which demonstrates that multi-dim based MTSA modeling both pairwise and global dependencies is more expressive than dot-product based multi-head self-attention. Although the results do not improve state-of-the-art BLEU value of machine translation task, the purpose of this experiment to verify the effectiveness of MTSA in contrast to dot-product based multi-head self-attention is accomplished. In conclusion, MTSA is highly parallelizable with more expressive power since it efficiently captures the pairwise dependency at token level, but delicately models the global dependency at feature level, and distributes computations to multiple heads, each equipped with a distinct positional mask. These lead to a sweet spot of the trade-off between performance and efficiency, and make MTSA as memory-efficient as CNN and scalable to long sequences but outperform previous (and even multi-dim) self-attention mechanisms in terms of prediction quality. The experiments conducted on nine NLP tasks verify that the MTSA can reach state-of-the-art performance with appealing efficiency. This research was funded by the Australian Government through the Australian Research Council (ARC) under grants 1) LP160100630 partnership with Australia Government Department of Health and 2) LP150100671 partnership with Australia Research Alliance for Children and Youth (ARACY) and Global Business College Australia (GBCA). We also acknowledge the support of NVIDIA Corporation and MakeMagic Australia with the donation of GPUs.
abstain abstain objective other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective other method abstain objective abstain abstain abstain abstain abstain other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain other abstain abstain other abstain abstain abstain abstain abstain other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain other other