The idea of subword-based word embeddings has been proposed in the literature, mainly for solving the out-of-vocabulary (OOV) word problem observed in standard word-based word embeddings. In this paper, we propose a method of reconstructing pre-trained word embeddings using subword information that can effectively represent a large number of subword embeddings in a considerably small fixed space. The key techniques of our method are twofold: memory-shared embeddings and a variant of the key-value-query self-attention mechanism. Our experiments show that our reconstructed subword-based embeddings can successfully imitate well-trained word embeddings in a small fixed space while preventing quality degradation across several linguistic benchmark datasets, and can simultaneously predict effective embeddings of OOV words. We also demonstrate the effectiveness of our reconstruction method when we apply them to downstream tasks 1 . Pre-trained word embeddings (or embedding vec-tors), especially those trained on a vast amount of text data, such as the Common Crawl (CC) corpus 2 , are now considered as highly beneficial, fundamental language resources. Typical examples of large, well-trained word embeddings are those trained on the CC corpus with 600 billion tokens by fastText (Bojanowski et al., 2017) and with 840 billion tokens by GloVe (Pennington et al., 2014), which we refer to as fastText.600B 3 and GloVe.840B 4 , respectively. In fact, we often 1 Our code and reconstructed subword-based word embeddings trained from GloVe.840B and fastText.600B are available: https://github.com/losyer/ compact_reconstruction 2 http://commoncrawl.org 3 https://fasttext.cc/docs/en/ english-vectors.html 4 https://nlp.stanford.edu/projects/ glove/ leverage such word embeddings to further improve the task performance of many natural language processing (NLP) tasks, such as constituency parsing (Suzuki et al., 2018; Gomez-Rodrguez and Vilares, 2018), discourse parsing (Yu et al., 2018), semantic parsing (Groschwitz et al., 2018; Dong and Lapata, 2018), and semantic role labeling (Strubell et al., 2018). Despite their significant impact on the NLP community, well-trained word embeddings still have several disadvantages. In this paper, we focus on two issues surrounding well-trained word embeddings: i) the massive memory requirement and ii) the inapplicability of out-of-vocabulary (OOV) words. It is crucial to address such issues, especially when applying them to real-world open systems. The total number of embeddings (i.e., the total memory requirement of such word embeddings) often becomes unacceptably large, especially in limited-memory environments, including GPUs, since the vocabulary size is more than 2 million words, which require at least 2 gigabytes (GB) of memory for storage. One possible solution is to merely discard (less important) words from the vocabulary, which can straightforwardly reduce the memory requirement. However, such a naive method can cause another well-known drawback regarding the inapplicability of OOV words. The applicability of OOV words is highly desirable in real systems since input words can be uncontrollably diverse. Therefore, there is a trade-off between the number of embedding vectors and the applicability of OOV words; thus, our goal is to investigate and develop a method that simultaneously has less memory requirement and high applicability of OOV words, which are both desirable properties for word embeddings in real-world open systems. Recently, methods that leverage subword information have been proposed and have become popular for overcoming the OOV word issue. Conceptually, the subword-based approach can cover all the words that can be constructed by a combination of subwords. Thus, the subword-based approach can greatly mitigate (or solve) the OOV word issue. We extend this approach to simultaneously enabling a reduction in the total number of embedding vectors through the reconstruction of word embeddings by subwords. The key techniques of our approach are twofold: memory-shared embeddings and a variant of the key-value-query (KVQ) self-attention mechanism (Vaswani et al., 2017). That is, our approach reconstructs well-trained word embeddings using a limited number of embedding vectors that are shared by all the subwords with an effective weighting calculated by the self-attention mechanism. In our experiments, we show that our reconstructed subword-based embeddings can successfully imitate well-trained word embeddings, such as fastText.600B and GloVe.840B , in a small fixed space while preventing quality degradation across several linguistic benchmark datasets from word similarity and analogy tasks. We also demonstrate the effectiveness of our reconstructed embeddings for representing the embeddings of OOV words. Lastly, we confirm the performance of our reconstructed embeddings on several downstream tasks from the named entity recognition task and the textual entailment task. The OOV word issue is one of the widely discussed topics in word embedding research, which several researches have recently attempted to solve. For example, methods that leverage subword information, such as character N -grams (in-cluding character unigrams) (Bojanowski et al., 2017; Pinter et al., 2017; Zhao et al., 2018) and morphological features (Luong et al., 2013), have recently been discussed as means of constructing word embeddings that consider the applicability of OOV words. Moreover, Pilehvar and Collier (2017) have proposed a method called SemLand , which induces OOV word embeddings by leveraging external resources. Bahdanau et al. (2017) and Herbelot and Baroni (2017) have also proposed methods that estimate OOV word embeddings using an additional LSTM and leveraging a small additional dataset, respectively. Among them, the study most closely related to ours is that of Zhao et al. (2018). Their basic idea is to reconstruct each pre-trained word embedding using a bag-of-character N -grams. We refer to their method as  BoS '. The motivation for reconstructing pre-trained word embeddings and utilizing character N -grams in our approach is substantially the same, however, an essential difference from BoS is that we additionally consider jointly reducing the total number of embedding vectors. Another study that shares the same motivation and goal is that of Pinter et al. (2017). Their method, referred to as MIMICK , utilizes only character information instead of character N -grams by mixing it with more sophisticated neural networks, i.e., LSTM (Hochreiter and Schmidhuber, 1997). MIMICK can produce a more compact model than the original word embeddings. The important difference between their method and ours is that our method only consists of the subword embeddings, whereas their method consists of the character embeddings and several transformation matrices for calculating LSTMs. We compare their method with ours in our experiments and empirically show the effectiveness of our approach. Moreover, Bojanowski et al. (2017) have proposed a method called fastText , which also incorporates character N -gram embeddings in addition to word embeddings. However, they did not explicitly prove the effectiveness of OOV word embeddings. Thus, it is still unclear how well the combination of character N -grams can reconstruct appropriate embeddings for OOV words. In addition, their method trains word embeddings from a corpus, which is not a reconstruction setting we discuss in this paper. Therefore, their method is orthogonal to ours. We often aim to reduce memory consumption of word embeddings in the real-world since they require relatively large memory. Suzuki and Nagata (2016) proposed a parameter reduction method for word embeddings by using machine learning techniques. Our method can also be interpreted as a kind of parameter reduction method based on the subword features. However, their method only considers the model shrinkage, and does not utilize any subword information nor consider the OOV issue. To summarize, none of the previous studies have attempted to simultaneously achieve a smaller number of embedding vectors and higher applicability of OOV words. Thus, in this paper, we report the first attempt to investigate how we can simultaneously achieve them. Additionally, deep contextualized pre-trained language models, such as ELMo (Peters et al., 2018), have recently been proposed as alternatives to the pre-trained word embeddings to further improve task performances. However, ELMo still takes advantage of Glove.840B to achieve its state-of-the-art performance. This fact implies that we can still combine the pre-trained word embeddings with strong pre-trained language models; thus, the importance of word embeddings in the literature remains unchanged even though stronger pre-trained models have been established. In this section, we explain a formal task definition that we tackle in this paper. Notation rules : In this paper, we use the following notation rules unless otherwise specified. First, a lower-case bold letter, e.g., v and e , represents a column vector, and an upper-case bold letter, e.g., V and E , represents a matrix. Then, (cid:107) v (cid:107) p represents L p -norm of the given vector v . Next, let a lower-case letter, e.g., z or i , be a scalar variable or index, and an upper-case letter, e.g., C or H , indicate a scalar but hyper-parameter during the training. Here, we introduce the notation V [ i ] to represent the i -th column vector in the matrix V to simplify the representation. Moreover, an upper-case letter in a calligraphy form, e.g., W and S , denotes a set, and the absolute value of a set, such as |W| and |S| , indicates the number of instances in the corresponding set. Finally, the Greek letter, such as  and  , indicates a function. Words and their embedding : Let W be a vocabulary, i.e., a set of words. Let  (  ) represent a mapping function from a word to the corresponding index of the word, namely,  (  ) : W  I w where I w = { 1 , . . . , |W|} . In this paper, we always assume that  (  ) is a bijective function; thus, each word has its own unique index between 1 and |W| . This also implies that the relation |W| = |I w | always holds. Let e w be a D -dimensional embedding vector for the word w  W , and let E denote an embedding matrix for all words in W , where E = RD |I w | . Then, we assume that the following relation always holds between e w and E : e w = E [ z e ] where z e =  ( w ) . Therefore, the i -th column vector in the matrix E represents the word embedding of the corresponding word w that satisfies i =  ( w ) . Subwords and their embedding : Let S be a vocabulary for all pre-defined subwords obtained from the words in W . Let  v (  ) represent a mapping function from a subword to the corresponding index of the subword, that is,  v (  ) : S  I s where I s = { 1 , . . . , |S|} . Similar to  in Eq. 1,  v (  ) is generally defined as a bijective function 5 . In this case, each subword has its own unique index between 1 and |S| , and the relation |S| = |I s | always holds. Here, we introduce v s as a D -dimensional vector for the subword s  S and V as an embedding matrix for all subwords in S , where V = RD |I s | . Then, we also assume the following relation between v s and V : v s = V [ z v ] where z v =  v ( s ) . Word to subword mapping : Additionally, we introduce a (abstract) function  (  ) that maps a word w  W to a list of subwords contained in the word w . We can define  (  ) in detail from several choices. For example, if we define  (  ) to extract all the character bi-grams appearing in a given word and w = higher', then we obtain a list of total seven distinct subword indices of  (cid:104) w (cid:105) h', hi', ig', gh', he', er', r (cid:104) /w (cid:105) ' as the return value of  ( w ) , where  (cid:104) w (cid:105) ' and  (cid:104) /w (cid:105) ' are special characters that represent the beginning and end of a word, respectively. Conceptually, we aim to reconstruct all the embeddings in E using V and a pre-defined subword mixing function  (  ) . Formally, our reconstruction problem is represented as a minimization problem of the following form:  V = arg min V (cid:8) ( E , V ,  ) (cid:9) , (5) 5 We redefine  v as a surjective function in Section 5.2. where (  ) is a loss function used to calculate the total reconstruction loss between E and V . As a brief summary, our goal is to find  V at which the loss function (  ) is minimized from the machine learning perspective. Note that the previous study, i.e., BoS , also utilized the above formulation for the reconstruction problem. Moreover, MIMICK can also be considered to utilize this formulation if V consists of all the single characters. Subword mixing function  (  ) : The role of the function  (  ) is to calculate an alternative embedding of the word embedding e w using a list of subwords contained in the given word w . One of the most popular definitions of  (  ) is to simply sum-up all the obtained subwords as follows:  sum ( V , w ) = (cid:88) s   ( w ) v s . In fact, a subword mixing function of this form was utilized in the previous studies, such as fastText and BoS . Loss function (  ) : First, to improve readability, we introduce  v w as a short notation of  ( V , w ) , namely  v w =  ( V , w ) . There are also several possible choices for the definition of the loss function (  ) . Here, we consider utilizing a squared loss function  lsq , which can be written as the summation of the squared losses over an individual embedding vector e w : ( E , V ,  ) = (cid:88) w W C w (cid:13)(cid:13) e w   v w (cid:13)(cid:13) 2 2 , (7) where C w is a weight factor for each word. Intuitively, (  ) is used to calculate the weighted sum of the L 2 -norm distances between the reference vector e w and a vector calculated by the subword mixing function  (  ) . In this section, we briefly explains the background that we still need to consider the number of embedding vectors in the subword-based approach. Table 1 shows the statistics of the total number of embedding vectors and the total memory requirement in several different settings. As shown in row (a) , the original fastText.600B word embeddings consist of 2 million words, which require 2.2 GB to store them since each word has a D = 300 dimensional vector. If we consider using all the character N -grams obtained from all ID Setting # of vecs mem. the words (row (e) ), then surprisingly, the memory requirement becomes approximately 25GB, which is too large for practical use. Therefore, it is crucial to technically reduce the memory requirement. A practical approach is to partially take advantage of a certain range of smaller N -grams, such as N = 1 to 3 (row (b) ) or N = 3 to 6 (row (c) ). However, smaller subword settings, such as row (b) , might markedly degrade the performance from the original word embeddings. Therefore, it is necessary to discover a better balance between the memory requirement (or the total number of embedding vectors) and performance. In this section, we describe several modifications to simultaneously achieve the purpose of smaller number of embedding vectors but higher performance with the applicability of OOV words. To do so, we incorporate several techniques in the baseline word embedding reconstruction approach explained in Section 3. Roughly speaking, we enhance the mapping function  v (  ) and the subword mixing function  (  ) . We take advantage of the topF frequent subwords that can be counted from the words in vocabulary W as a subword vocabulary instead of all possible subwords S . Let SF represent the set of the top-F frequent subwords, where SF  S . Then, we define a new mapping function  v,F (  ) as follows:  v,F (  ) : SF I s,F where I s,F = { 1 , . . . , |S F |} . deFigure 1: Intuitive idea of our memory-shared embeddings by hashing. Here, we assume H (cid:48) < H . scribed in Section 3.1. Again, this means that each subword has its own unique subword index. Here, we modify  v (  ) as a surjective function by introducing the following new mapping function  v,H (  ) as a replacement of  v (  ) in Eq. 3:  v,H (  ) : S I s,H where I s,H = { 1 , . . . , H } . Here, we assume H < |S| . This mapping function  v,H (  ) implies that each subword is mapped to an index, but the index is not unique and may be shared between other subwords. Therefore, the number of subword embeddings can also be reduced to H by sharing the embeddings; thus, the subword embedding matrix V can also be reduced from V = RD |S| to V = RD  H . Here, if we assume the relation H (cid:28) |S| , then we can greatly reduce the total embedding size. There are several possible choices for the definition of the mapping function  v,H (  ) . We select a simple hash function for  v,H (  ) . This means that subword embeddings are randomly shared over the subwords. One large merit of using simple hash functions is that we require no external mapping structure of every (subword, subword index) pair, which also matches our goal of reducing the memory requirement in actual use cases. Figure 1 illustrates an intuitive idea of memory-shared embeddings by hashing. Also  v,F (  ) and  v,H (  ) can be combined step by step. First, we reduce the subword vocabulary S to topF frequent subwords SF as described in Section 5.1. Second, we apply our memory sharing method to only SF in contrast to applying it to S Figure 2: Illustration of how our KVQ self-attention operation calculates each word embedding. in Section 5.2. Here, we define a new mapping function  v,FH (  ) as follows:  v,FH (  ) : SF I s,H where I s,H = { 1 , . . . , H } . Previous researches such as fastText and BoS treat  ( V , w ) as a summation of all subword embeddings described by Eq. 6. However, the summation is less expressive and it may lack capability in a memory-sharing setting since subwords share their embeddings randomly. One possible improvement is to handle the importance of each subword based on a given word during the calculation of ( V , w ) . A simple approach to deal with this phenomenon is to incorporate a context-dependent weighting factor for each subword in a given word. Thus, we consider the following subword mixing function  kvq ( V , w ) as:  kvq ( V , w ) = (cid:88) s   ( w ) a s,w v s (11) where a s,w represents a context-dependent weighting factor of the subword s , where the context here means all the subwords obtained from word w . To calculate a s,w , we first introduce k s and q s , which are similarly defined to v s in Eq. 4, namely, k s = V [ z k ] , where z k =  k ( s ) and q s = V [ z q ] , where z q =  q ( s ) . Similar to  v (  ) in Eq. 3,  k (  ) and  q (  ) are two distinct mapping functions that map a given subword s into a subword index. Then, we introduce a key-value-query (KVQ) self-attention operation inspired by data number of OOV data abbre. Transformer (Vaswani et al., 2017), that is, a s,w = exp( Z  q  k s ) (cid:80) s (cid:48)   ( w ) exp( Z  q  k s (cid:48) ) , (12) where Z is a scaling hyper-parameter. and  q = (cid:80) s   ( w ) q s . Figure 2 illustrates how our KVQ self-attention operation calculates each word embedding. Evaluation data : Table 2 shows a summary of the evaluation datasets used in our experiments. We conducted experiments on well-studied linguistic benchmark datasets, i.e., nine for word similarity ( WordSim ) tasks and two for word analogy ( Analogy ) tasks. In this evaluation, we discarded data in the evaluation datasets if at least one of the words in the data was an OOV word. Note that this is the standard evaluation criterion used in the previous studies. By following this criterion, we investigate the effectiveness in terms of model shrinkage since we can fairly compare the performance with the original (word-based) word embeddings. Pre-trained word embeddings : For the reconstruction target, we selected fastText.600B 6 . Note that it achieved the state-of-the-art performance on the WordSim and Analogy datasets (Bo-janowski et al., 2017). The hyper-parameters D = 300 and |W| = 2 M were automatically obtained from the properties of fastText.600B . Hyper-parameters for training : We took advantage of a N -grams' range of N = 3 to 30 . We adopted Adam (Kingma and Ba, 2014) as our optimization algorithm to minimize Eq. 5. We set the following hyper-parameters for Adam:  = 0 . 0001 ,  1 = 0 . 9 ,  2 = 0 . 999 , and (cid:15) = 1  10  8 . We leveraged a mini-batch training, whose size was 200, and trained each method for 300 epochs. For C w in Eq. 7, we utilized the occurrence information calculated from a large external corpus. Then, we set Z =  D for all experiments. Comparison : We compared the following five distinct settings of subword-based reconstruction of word embeddings. 1. SUM-F : Select  v,F (  ) in Eq. 8 (Section 5.1) for the subword mapping function and  sum (  ) in Eq. 6 for the subword mixing function. 2. SUM-H : As in the first setting but substitute  v,F (  ) with  v,H (  ) in Eq. 9 (Section 5.2). 3. KVQ-H : As in the second setting but substitute  sum (  ) in Eq. 6 with  kvq ( V , w ) in Eq. 11 (Section 5.4). 4. SUM-FH : As in the second setting but substitute  v,H (  ) with  v,FH (  ) in Eq. 10 (Sec-tion 5.3). 5. KVQ-FH : As in the third setting but substitute  v,H (  ) with  v,FH (  ) in Eq. 10 (Section 5.3). 6 Additionally, we also conducted the same experiments using GloVe.840B instead of fastText.600B . See Appendix A for the results. curves for WordSim and Analogy, respectively. Each plot is the macro-average of Spearman's rho (WordSim) or micro-average of accuracy (Anal-ogy) of all evaluation datasets. Moreover, Table 3 shows the statistics of each setting and Table 4 show a summary of the detailed results for all datasets. Overall, we observed a consistent tendency that the methods using KVQ obtained the best performance compared with the methods using SUM. Notably, KVQ-H significantly outperformed SUM-F , SUM-H and SUM-FH by more than 10 points in terms of the micro-average accuracy of all Analogy datasets when the model size was 0 . 5 M. The difference between KVQ-H and SUM-H shows that the approach of memory-shared embeddings is particularly useful when we combined it with the KVQ operation. Moreover, in some cases, we observed that KVQ-H achieved the performance of the original word embeddings fastText.600B when H = 0 . 5 M. This means that KVQ-H with H = 0 . 5 M with successfully reduced the model size nearly fourfold compared with the original word embeddings while maintaining the original performance. Time efficiency of SUM and KVQ : We investigated the difference in time efficiency between SUM and KVQ. We calculated the time computing word embeddings from subword embeddings using each operation. SUM and KVQ took 5 . 7  10  6 and 1 . 6  10  5 seconds per word, respectively, i.e., KVQ took 2.8 times longer than SUM. However, the calculation speed per word is sufficiently high to be negligible in the real applications since other operations such as calculating deep neural networks may take much longer. Evaluation data : We used the identical nine WordSim datasets used in Section 6.1. Preparation of training data : As we showed in Table 2, the numbers of OOV problems for fastText.600B and GloVe.840B are indeed very small. This is because their vocabulary sizes exceeds 2 million words, and the words contained in the evaluation datasets tend to be non-rare words' in general. Therefore, it is difficult to precisely evaluate the effectiveness of the estimation of OOV word embeddings. To overcome this difficulty, we artificially made our reconstruction problem much more difficult, namely, we discarded the words contained in the evaluation datasets from the vocabulary W for training. This means that all the problems in the evaluation datasets now became OOV problems. In other words, the number of OOV data in Table 2 in this setting always matches to the evaluation data size, such as 2034 for RW. Other settings : We used the same experimental settings as used in Section 6.1 unless otherwise specified. For example, we used fastText.600B as the reconstruction target and the same training hyper-parameters. Table 5 shows the results of the (synthetic) OOV word experiments. First, we observed that the performance of the Random baseline's was nearly equal to zero across all the datasets. This means that there is no correlation between the Random and human-annotated scores. Importantly, the performances of KVQ-FH , SUM-FH and SUM-F were significantly improved by 44-53 points from that of Random . This result indicates that KVQ-FH , SUM-FH and SUM-F successfully predicted the OOV word embeddings. However, we also observed no significant difference between KVQ-FH , SUM-FH and SUM-F for both the H = 0 . 5 M and H = 0 . 2 M settings. As we discussed in Section 2, several closely related methods have also tackled to solve the OOV word issue, such as MIMICK and BoS . We aim to directly compare our approach with these methods to investigate whether it can outperform them. However, these methods (or available authors' codes) do not work on the large-vocabulary settings employed in Sections 6.1 and 6.2. Thus, as an alternative, we strictly followed the experimental settings described in (Zhao et al., 2018) and compared the performance under fair conditions. Evaluation data : In their evaluation setting, they evaluated the OOV word performance over RW shown in Table 2. They included all the words appearing in RW as the evaluation data, in contrast to discarding the OOV data as in Section 6.1. Pre-trained word embeddings : The target word embeddings for the reconstruction were the embeddings trained on Google News with 100 billion tokens 7 that were pre-cleaned by (Zhao et al., 2018). The resultant embeddings consist of 0.16M lower-cased word embeddings. Comparison : We compared our approach with the following related methods: 1. Random : the performance when we used random vectors for OOV words. 2. MIMICK 8 (Pinter et al., 2017). 3. BoS 9 (Zhao et al., 2018). 7 https://code.google.com/archive/p/ word2vec/ 8 https://github.com/yuvalpinter/Mimick 9 https://github.com/jmzhao was relatively very small compared with that in our experiments, i.e., 2M vs 0.16M. Results : Table 6 shows a comparison with the related methods. All our reconstruction methods outperformed BoS , which was the previous state-of-the-art method, with substantial improvements by 2-6 points. Moreover, KVQ-FH achieved the best performance in this comparison. To investigate the effectiveness of our recon-stucted embeddings in downstream tasks, we evaluated them in the named entity recognition (NER) and the textual entailment (TE) tasks. Evaluation data : We used the CoNLL 2003 dataset (Tjong Kim Sang and De Meulder, 2003) for an NER experiment and the Stanford Natural Language Inference (SNLI) dataset (Bowman et al., 2015) for a TE experiment. Other settings : We used fastText.600B and GloVe.840B as the target word embeddings for the reconstruction. For our reconstruction embeddings, we calculated the embeddings of all the words in the datasets, thus there exist no OOV words when using our methods. We used AllenNLP 10 to train base NER and TE models. We basically used the provided hyper-parameter values in their repository for both training and testing. Additionally, we added one hyper-parameter K to re-scale embeddings (i.e., multiply all the elements in the embeddings by K ) 10 https://allennlp.org/ since we learned that the re-scaling may significantly affect the overall performance of downstream tasks in certain situation. We search K from [1 , 5 , 10 , 20 , 50 , 100] on the validation set of each dataset. Tables 7 and 8 show the comparison between the original (large) embeddings and our reconstructed (small) embeddings. When H = 0 . 5 M, the performances of our reconstructed embeddings are equivalent to or even better than the ones of the original embeddings. One might be surprised the improved results by KVQ-FH since the model sizes of KVQ-FH were relatively very small comparing with the original embeddings. However, this may be a reasonable observation since our method additionally offered the embeddings of OOV words that cannot be handled by the original embeddings. Moreover, even when H = 0 . 2 M, i.e. the model size was approximately ten times smaller, the degradation was less than 1.0, which is considered to be sufficiently acceptable for real-world systems if we can significantly reduce the model (system) size. We discussed and investigated an approach that reconstructs subword-based word embeddings in a reduced memory space. We demonstrated that memory-shared embeddings with the KVQ self-attention operation significantly outperformed the conventional summation-based approach, such as BoS . Moreover, our best setting successfully reduced the number of embedding vectors to approximately ten times smaller than that of the original word embeddings while maintaining an acceptable performance loss on the downstream tasks. We also confirmed the effectiveness of our approach in terms of the applicability of OOV words. We believe that our reconstructed subword-based word embeddings can be better alternatives of fastText.600B and GloVe.840B because they require less memory requirement and have high applicability of OOV words. This work was supported by JSPS KAKENHI Grant Number 15H01702 and JST CREST Grant Number JPMJCR1513, Japan.
abstain objective abstain result objective abstain method other abstain method abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain objective abstain method result objective method other other other other other other abstain objective other other other abstain abstain other other other method other objective other abstain other other objective other other method method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method method abstain method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain result abstain abstain abstain abstain abstain abstain abstain other abstain result abstain abstain abstain abstain abstain abstain objective objective method method method other