Abstract The use of subword-level information (e.g., characters, character n-grams, morphemes) has become ubiquitous in modern word representation learning. Its importance is attested especially for morphologically rich languages which generate a large number of rare words. Despite a steadily increasing interest in such subword-informed word representations , their systematic comparative analysis across typologically diverse languages and different tasks is still missing. In this work, we deliver such a study focusing on the variation of two crucial components required for subword-level integration into word representation models: 1) segmentation of words into subword units, and 2) subword composition functions to obtain final word representations. We propose a general framework for learning subword-informed word representations that allows for easy experimentation with different segmentation and composition components, also including more advanced techniques based on position embeddings and self-attention. Using the unified framework, we run experiments over a large number of subword-informed word representation configurations (60 in total) on 3 tasks (gen-eral and rare word similarity, dependency parsing, fine-grained entity typing) for 5 languages representing 3 language types. Our main results clearly indicate that there is no one-size-fits-all configuration, as performance is both languageand task-dependent. We also show that configurations based on unsupervised segmentation (e.g., BPE, Morfessor) are sometimes comparable to or even outperform the ones based on supervised word segmentation. Word representations are central to a wide variety of NLP tasks (Collobert et al., 2011; Chen and Manning, 2014; Jia and Liang, 2016; Ammar et al., 2016; Goldberg, 2017; Peters et al., 2018; Kudo, 2018, inter alia ). Standard word representation models are based on the distributional hypothesis (Harris, 1954) and induce representations from large unlabeled corpora using word co-occurrence statistics (Mikolov et al., 2013; Pennington et al., 2014; Levy and Goldberg, 2014). However, as pointed out by recent work (Bojanowski et al., 2017; Vania and Lopez, 2017; Pinter et al., 2017; Chaudhary et al., 2018; Zhao et al., 2018), mapping a finite set of word types into corresponding word representations limits the capacity of these models to learn beyond distributional information, which leads to several fundamental limitations. The standard approaches ignore the internal structure of words, that is, the syntactic or semantic composition from subwords or morphemes to words, and are incapable of parameter sharing at the level of subword units. Assigning only a single vector to each word causes the data sparsity problem, especially in resource-poor settings where huge amounts of training data cannot be guaranteed. The issue is also prominent for morphologically rich languages (e.g., Finnish) with productive morphological systems that generate a large number of infrequent/rare words (Gerz et al., 2018). Although potentially useful information on word relationships is hidden in their internal subword-level structure , 1 subword-agnostic word representation models do not take these structure features into account and are effectively unable to represent rare words accurately, or unseen words at all. Therefore, there has been a surge of interest in subword-informed word representation architectures aiming to address these gaps. A large number of architectures has been proposed in related research, and they can be clustered over the two main axes (Lazaridou et al., 2013; Luong et al., 2013; 1 For example, nouns in Finnish have 15 cases and 3 plural forms; Spanish verbs may contain over 40 inflected forms, sharing the lemma and taking up standard suffixes. Qiu et al., 2014; Cotterell and Schutze, 2015; Wi-eting et al., 2016; Avraham and Goldberg, 2017; Vania and Lopez, 2017; Pinter et al., 2017; Cotterell and Schutze, 2018). First, the models differ in the chosen method for segmenting words into subwords . The methods range from fully supervised approaches (Cotterell and Schutze, 2015) to e.g. unsupervised approaches based on BPE (Heinzerling and Strube, 2018). Second, another crucial aspect is the subword composition function used to obtain word embeddings from the embeddings of each word's constituent subword units. Despite a steadily increasing interest in such subword-informed word representations, their systematic comparative analysis across the two main axes, as well as across typologically diverse languages and different tasks is still missing. 2 In this work, we conduct a systematic study of a variety of subword-informed word representation architectures that all can be described by a general framework illustrated by Figure 1. The framework enables straightforward experimentation with prominent word segmentation methods (e.g., BPE, Morfessor, supervised segmentation systems) as well as subword composition functions (e.g., addition, self-attention), resulting in a large number of 2 A preliminary study of Vania and Lopez (2017) limits its focus on the use of subwords in the language modeling task. different subword-informed configurations . 3 Our study aims at providing answers to the following crucial questions: Q1) How generalizable are subword-informed models across typologically diverse languages and across different downstream tasks? Do different languages and tasks require different configurations to reach peak performances or is there a single best-performing configuration? Q2) How important is it to choose an appropriate segmentation and composition method? How effective are more generally applicable unsupervised segmentation methods? Is it always better to resort to a supervised method, if available? Q3) Is there a difference in performance with and without the full word representation? Can more advanced techniques based on position embeddings and self-attention yield better task performance? We evaluate subword-informed word representation configurations originating from the general framework in three different tasks using standard benchmarks and evaluation protocols: 1) general and rare word similarity and relatedness, 2) dependency parsing, and 3) fine-grained entity typing for 5 languages representing 3 language families (fusional, introflexive, agglutinative). We show that different tasks and languages indeed require diverse subword-informed configurations to reach peak performance: this calls for a more careful language-and task-dependent tuning of configuration components. We also show that more sophisticated configurations are particularly useful for representing rare words, and that unsupervised segmentation methods can be competitive to supervised segmentation in tasks such as parsing or fine-grained entity typing. We hope that this paper will provide useful points of comparison and comprehensive guidance for developing next-generation subword-informed word representation models for typologically diverse languages. The general framework for learning subword-informed word representations, illustrated by Figure 1, is introduced in  2.1. We then describe its main components: segmentation of words into subword units (  2.2), subword and position embeddings (  2.3), and subword embedding composition 3 Following a similar work on subword-agnostic word embedding learning (Levy et al., 2015), our system design choices resulting in different configurations can be seen as a set of hyper-parameters that also have to be carefully tuned for each language and each application task. where  ( w ) is a deterministic function that segments w into an ordered sequence of its constituent subword units S w = ( s w i ) n 1 , with s w i  S being a subword type from the subword vocabulary S of size |S| . Optionally, some segmentation methods can also generate a sequence of the corresponding morphotactic tags T w = ( t w i ) n 1 . Alone or together with T w , S w is embedded into a sequence of subword representations S w = ( s w i ) n 1 from the subword embedding matrix W s  R |S| d , where d is the dimensionality of subword embeddings. Another optional step is to obtain a sequence of position embeddings P w = ( p w i ) n 1 : they are taken from the position embedding matrix W p  R p  d , where p is the maximum number of the unique positions. P w can interact with S w to compute the final representations for subwords R w = ( r w i ) n 1 (Vaswani et al., 2017). f  is a composition function taking R w as input and outputting a single vector w as the word embedding of w . For the distributional word-level training, similar to prior work (Bojanowski et al., 2017), we adopt the standard skip-gram with negative sampling ( SGNS ) (Mikolov et al., 2013) with bag-of-words contexts. However, we note that other distributional models can also be used under the same framework. Again, following Bojanowski et al. (2017), we calculate the word embedding w t  R d for each target word w t using the formulation from Eq. (1), and parametrize context words with another word embedding matrix W c  R |V| d , where |V| is the size of word vocabulary V . Supervised Morphological Segmentation We use CHIPMUNK (Cotterell et al., 2015) as a representative supervised segmentation system, proven to provide a good trade-off between accuracy and speed. 4 It is based on semi-Markov conditional ran-4 http://cistern.cis.lmu.de/chipmunk   ( dishonestly ) CHIPMUNK ( dis , honest , ly ) ( prefix , root , suffix ) Morfessor ( dishonest , ly ) BPE ( dish , on , est , ly ) Table 1: Segmentations of the word dishonestly . dom fields (Sarawagi and Cohen, 2005). For each word, apart from generating S w , it also outputs the corresponding morphotactic tags T w . 5 In  2.3 we discuss how to incorporate information from T w into subword representations. Morfessor Morfessor (Smit et al., 2014) denotes a family of generative probabilistic models for unsupervised morphological segmentation used, among other applications, to learn morphologically-aware word embeddings (Luong et al., 2013). BPE Byte Pair Encoding (BPE; Gage (1994)) is a simple data compression algorithm. It has become a de facto standard for providing subword information in neural machine translation (Sennrich et al., 2016). The input word is initially split into a sequence of characters, with each unique character denoted as a byte. BPE then iteratively replaces the most common pair of consecutive bytes with a new byte that does not occur within the data, and the number of iterations can be set in advance to control the granularity of the byte combinations. An example output for all three methods is shown in Table 1. Note that a standard practice in subword-informed models is to also insert the entire word token into S w (Bojanowski et al., 2017). 6 This is, however, again an optional step and we evaluate configurations with and without the inclusion of the word token in S w . The next step is to encode S w (or the tuple ( S w , T w ) for CHIPMUNK) to construct a sequence of subword representations S w . Each row of the subword embedding matrix W s is simply defined as the embedding of a unique subword. For CHIPMUNK, we define each row in W s as the concatenation of the subword s and its predicted tag t . We also test 5 In our experiments, we use only basic information on affixes such as prefixes and suffixes, and leave the integration of fine-grained information such as inflectional and derivational affixes as future work. 6 We only do the insertion if | S w | > 1 . For CHIPMUNK, a generic tag word is added to the sequence T w . CHIPMUNK configurations without the use of T w to analyze its contribution. 7 After generating S w , an optional step is to have a learnable position embedding sequence P w further operate on S w to encode the order information. Similar to W s , the definition of the position embedding matrix W p also varies: for Morfessor and BPE, we use the absolute positions of subwords in the sequence S w , whereas for CHIPMUNK morphotactic tags are encoded directly as positions. Finally, following prior work (Gehring et al., 2017; Mikolov et al., 2018), we use addition and element-wise multiplication between each subword vector s from S w and the corresponding position vector p from P w to compute each entry r for the final sequence of subword vectors R w : r = s + p or r = s (cid:12) p . A composition function f  is then applied to the sequence of subword embeddings R w to compute the final word embedding w . We investigate three composition functions: 1) addition, 2) single-head and 3) multi-head self-attention (Vaswani et al., 2017; Lin et al., 2017). 8 Addition is used in the original fastText model of Bojanowski et al. (2017), and remains a strong baseline for many tasks. However, addition treats each subword with the same importance, ignoring semantic composition and interactions among the word's constituent subwords. Therefore, we propose to use a self-attention mechanism, that is, a learnable weighted addition as the composition function on subword sequences. To the best of our knowledge, we are the first to apply a self-attention mechanism to the problem of subword composition. Composition Based on Self-Attention Our self-attention mechanism is inspired by Lin et al. (2017). It is essentially a multilayer feed-forward neural network without bias term, which generates a weight matrix for the variable length input R w : H w = tanh ( W h 1 R Tw ) (3) A w = softmax ( W h 2 H w ) (4) 7 The extra information on tags should lead to a more expressive model resolving subword ambiguities. For instance, the subword post in postwar and noun post are intrinsically different: the former is the prefix and the later is the root. Each row of A w is a weight vector for rows of R w , which models different aspects of semantic compositions and interactions. For the single-head self-attention, A w degenerates to a row vector as the final attention vector a w . For the multihead self-attention, we average the rows of A w to generate a w . 9 Finally, w is computed as the weighted addition of subword embeddings from R w : w = (cid:80) w n w 1 a w i r w i . We train different subword-informed model configurations on 5 languages representing 3 morphological language types: English ( EN ), German ( DE ), Finnish ( FI ), Turkish ( TR ) and Hebrew ( HE ), see Table 3. We then evaluate the resulting subword-informed word embeddings in three distinct tasks: 1) general and rare word similarity and relatedness, 2) syntactic parsing, and 3) fine-grained entity typing. The three tasks have been selected in particular as they require different degrees of syntactic and semantic information to be stored in the input word embeddings, ranging from a purely semantic task (word similarity) over a hybrid syntactic-semantic task of entity typing to syntactic parsing. Subword-Informed Configurations We train a large number of subword-informed configurations by varying the segmentation method  (  2.2), subword embeddings W s , the inclusion of position embeddings W p and the operations on W s (  2.3), and the composition functions f  (  2.4). The configurations are based on the following variations of 9 We have also experimented with adding an extra transformation layer over attention matrix to generate the attention vector, but without any performance gains. constituent components: (1) For the segmentation  , we test a supervised morphological system CHIPMUNK ( sms ), Morfessor ( morf ) and BPE ( bpe ). A word token can be optionally inserted into the subword sequence S w for all three segmentation methods ( ww ) or left out ( w); (2) We can only embed the subword s for morf and bpe , while with sms we can optionally embed the concatenation of the subword and its morphotactic tag s : t ( st ); 10 (3) We test subword embedding learning without position embeddings ( p), or we integrate them using addition ( pp ) or element-wise multiplication ( mp ); (4) For the composition function function f  , we experiment with addition ( add ), single head self-attention ( att ), and multi-head self-attention ( mtx ). Table 2 provides an overview of all components used to construct a variety of subword-informed configurations used in our evaluation. The variations of components from Table 2 yield 24 different configurations in total for sms , and 18 for morf and bpe . We use pretrained CHIPMUNK models for all test languages except for Hebrew, as Hebrew lacks gold segmentation data. Following Vania and Lopez (2017), we use the default parameters for Morfessor, and 10k merge operations for BPE across languages. We use available BPE models pre-trained on Wikipedia by Heinzerling and Strube (2018). 11 Two well-known word representation models, which can also be described by the general framework from Figure 1, are used as insightful baselines: the subword-agnostic SGNS model (Mikolov et al., 2013) and fastText ( FT ) 12 (Bojanowski et al., 2017). FT computes the target word embedding using addition as the composition function, while the segmentation is straightforward: the model simply generates all character n-grams of length 3 to 6 and adds them to S w along with the full word. Training Setup Our training data for all languages is Wikipedia. We lowercase all text and replace all digits with a generic tag # . The statistics of the training corpora are provided in Table 3. All subword-informed variants are trained on the same data and share the same parameters for the SGNS model. 13 Further, we use ADAGRAD (Duchi 10 Once st is applied, we do not use position embeddings anymore, because the morphotactic tags are already encoded in subword embeddings, i.e., st and pp are mutually exclusive. 11 https://github.com/bheinzerling/bpemb 12 https://github.com/facebookresearch/ fastText 13 We rely on the standard choices: 300 -dimensional sub-Typology Language #tokens #types Fusional English ( EN ) 600M 900K German ( DE ) 200M 940K Agglutinative Finnish ( FI ) 66M 600K Turkish ( TR ) 52M 300K Introflexive Hebrew ( HE ) 90M 410K Table 3: Statistics of our Wikipedia training corpora. et al., 2011) with a linearly decaying learning rate, and do a grid search of learning rate and batch size for each  on the German 14 WordSim-353 data set ( WS ; Leviant and Reichart (2015)). The hyperparameters are then fixed for all other languages and evaluation runs. Finally, we set the learning rate to 0.05 for sms and bpe , and 0.075 for morf , and the batch size to 1024 for all the settings. Word Similarity and Relatedness These standard intrinsic evaluation tasks test the semantics of word representations (Pennington et al., 2014; Bojanowski et al., 2017). The evaluations are performed using the Spearman's rank correlation score between the average of human judgement similarity scores for word pairs and the cosine similarity between two word embeddings constituting each word pair. We use Multilingual SimLex-999 ( SIMLEX ; Hill et al. (2015); Leviant and Reichart (2015); Mrksic et al. (2017)) for English, German and Hebrew, each containing 999 word pairs annotated for true semantic similarity. We further evaluate embeddings on FinnSim-300 ( FS 300) produced by Venekoski and Vankka (2017) for Finnish and AnlamVer ( AN ; Ercan and Yldz (2018)) for Turkish. We also run experiments on the WordSim-353 test set ( WS ; Finkelstein et al. (2002)), and its portions oriented towards true similarity ( WS-SIM ) and broader relatedness ( WS-REL ) portion for English and German. Finally, to analyze the importance of subword information for learning embeddings of rare words, we evaluate on the recently released CARD-660 dataset ( CARD ; Pilehvar et al. (2018)) for English, annotated for true semantic similarity. word and word embeddings, 5 training epochs, the context window size is 5, 5 negative samples, the subsampling rate of 10  5 , and the minimum word frequency is 5. 14 German has moderate morphological complexity among the five languages, so we think the hyperparameters tuned on it could be applicable to other languages. Dependency Parsing Next, we use the syntactic dependency parsing task to analyze the importance of subword information for syntactically-driven downstream applications. For all test languages, we rely on the standard Universal Dependencies treebanks (UD v2.2; Nivre et al. (2016)). We use subword-informed word embeddings from different configurations to initialize the deep biaffine parser of Dozat and Manning (2017) which has shown competitive performance in shared tasks (Dozat et al., 2017) and among other parsing models (Ma and Hovy, 2017; Shi et al., 2017; Ma et al., 2018). 15 We use default settings for the biaffine parser for all experimental runs Fine-Grained Entity Typing The task is to map entities, which could comprise more than one entity token, to predefined entity types (Yaghoobzadeh and Schutze, 2015). It is a suitable semi-semantic task to test our subword models, as the subwords of entities usually carry some semantic information from which the entity types can be inferred. For example, Lincolnshire will belong to /location/county as -shire is a suffix that strongly indicates a location. We rely on an entity typing dataset of Heinzerling and Strube (2018) built for over 250 languages by obtaining entity mentions from Wikidata (Vrandecic and Krotzsch, 2014) and their associated FIGER-based entity types (Ling and Weld, 2012): there only exists a one-to-one mapping between the entity and one of the 112 FIGER types. We randomly sample the data to obtain a train/dev/test split with the size of 60k/20k/20k for all languages. For evaluation we extend the RNN-based model of Heinzerling and Strube (2018), where they stacked all the subwords of entity tokens into a flattened sequence: we use the hierarchical embedding composition instead. For each entity token, we first compute its word embeddings with our subword configurations, 16 then feed the word embeddings of entity tokens to a bidirectional LSTM with 2 hidden layers of size 512, followed by a projection layer which predicts the entity type. 15 https://github.com/tdozat/Parser-v2 16 Although it is true that case information can be very important to the task, we conform to Heinzerling and Strube (2018) lowercasing all letters. To get a better grasp of the overall performance without overloading the tables, we focus on reporting two best configurations and the worst configuration for each task and language from the total of 60 configurations, except for Hebrew with 36 configurations, where there is no gold segmentation data for training sms model. We also analyze the effects of different configurations on different tasks based on language typology. The entire analysis revolves around the key questions Q1-Q3 posed in the introduction. The reader is encouraged to refer to the supplementary material for the complete results. Tables 4, 5, and 6 summarize the main results on word similarity and relatedness, dependency parsing and entity typing, respectively. In addition, the comparisons of different configurations across tasks and language types are shown in Figure 2 (as well as Figure 3 to 7 in the supplementary ma-terial). There, we center the comparison around two crucial components: segmentation and composition. The value in each pixel block is the percentage rank of the row configuration minus that of column configuration. We compute such percentage ranks by performing three levels of averaging over: 1) all related datasets for the same task; 2) all sub-configurations that entail the configuration in question; 3) all languages from the same language types. Q1. Tasks and Languages Regarding the absolute performance of our subword-informed configurations, we notice that they outperform SGNS and FT in 3/5 languages on average, and for 8/13 datasets on word similarity and relatedness. The gains are more prominent over the subword-agnostic SGNS model and for morphologically richer languages such as Finnish and Turkish. The results on the two other tasks are also very competitive, with strong performance reported especially on the entity typing task. This clearly indicates the importance of integrating subword-level information into word vectors. A finer-grained comparative analysis shows that best-performing configurations vary greatly across different languages. The comparative analysis across tasks also suggests that there is no single configuration that outperforms the others in all three tasks, although certain patterns in the results emerge. For instance, the supervised segmentation ( sms ) is very useful for word similarity and relatedness (seen in Figure 4). This result is quite intuitive: sms is trained according to the readily available gold standard morphological segmentations. However, sms is less useful for entity typing, where almost all best-performing configurations are based on morf (see also Figure 4). This result is also interpretable: morf is a conservative segmenter that captures longer subwords, and is not distracted by short and nonsensical subwords (like bpe ) that have no contribution to the prediction. 17 The worst configurations on word similarity are based on bpe : its aggressive segmentation often results in non-interpretable or nonsensical subwords which are unable to recover useful semantic information. Due to the same reason, the results in all three tasks indicate that the best configurations with bpe are always coupled with ww , and the worst are obtained with w(i.e., without the inclusion of the full word). The results on parsing reveal similar performances for a spectrum of heterogeneous configurations. In other words, while the chosen configuration is still important, its impact on performance of state-of-the-art dependency parsers (Kiperwasser and Goldberg, 2016; Dozat and Manning, 2017) is decreased, as such parsers are heavily parametrized multi-component methods (e.g., besides word embeddings they rely on biLSTMs, intra-sentence attention, character representations). 18 Therefore, a larger space of subword-informed configurations for word representations leads to optimal or near-optimal results. However, sms seems to yield highest scores on average in agglutinative languages (see also Figure 2). Figure 2 clearly demonstrates that, apart from entity typing heatmaps (the third row) which show very similar trends over different language types, the patterns for different tasks and language types tend to vary in general. Similarly, other figures in the supplementary material also show diverging trends across languages representing different language types. 17 For example, sms and bpe both split Val-berg ( /location/city ) and Robert Valberg ( /person/actor ) with a suffix berg. Since berg could represent both a place or a person, it is not useful alone as a suffix to predict the correct entity type, whereas morf does not split the word and makes the prediction solely based on surrounding entity tokens. 18 We also experimented with removing token features such as POS tags and character embeddings in some settings, but we observed similar trends in the final results. segmentation and composition components. The crucial components for word similarity are sms and ww , and sms is generally better than morf and bpe in fusional and agglutinative languages (see Figure 4 and 7). The presence of ww is desired in this task as also found by Bojanowski et al. (2017): it enhances the information provided by the segmentation. As discussed before, the best configurations with bpe are always coupled with ww , and the worst with w. ww is less important for the more conservative morf where the information stored in ww can be fully recovered from the generated subwords. Interestingly, pp and mp do not have positive effects on this task for fusional and introflexive languages, but they seem to resonate well with agglutinative languages, and they are useful for the two other tasks (seen in Figure 6). In general, position embeddings have shown potential benefits in all tasks, where they selectively emphasize or filter subwords according to their positions. pp is extremely useful in entity typing for all languages, because it indicates the root position. Concerning composition functions, add still remains an extremely robust choice across tasks and languages. Surprisingly, the more sophisticated self-attention composition prevails only on a handful of datasets: compare the results with add vs. att and mtx . In fact, the worst configurations mostly use att and mtx (see also Figure 5). In sum, our results suggest that, when unsure, add is by far the most robust choice for the subword composition function. Further, morphotactic tags encoded in subword embeddings ( st ) seem to be only effective combined with self-attention in word similarity and relatedness. These findings call for further investigation in future work, along with the inclusion of finer-grained morphotactic tags into the proposed modeling framework. Further Discussion A recurring theme of this study is that subword-informed configurations are largely taskand language-dependent. We can extract multiple examples from the reported results affirming this conjecture. For instance, in fusional and agglutinative languages mp is critical to the model on dependency parsing, while for Hebrew, an introflexive language, mp is among the most detrimental components on the same task. Further, for Turkish word similarity bpe.ww outperforms sms.ww : due to affix concatenation in Turkish, sms produces many affixes with only syntactic functions that bring noise to the task. Interestingly, SGNS performs well in Hebrew on parsing and word similarity: it shows that it is still difficult for linear segmentation methods to capture non-concatenative morphology. Finally, fine-tuning subword-informed representations seems especially beneficial for rare word semantics: our best configuration outperforms FT by 0.111 on CARD , and even surpasses all the state-of-the-art models on the rare word similarity task, as reported by Pilehvar et al. (2018). We hope that our findings on the CARD dataset will motivate further work on building more accurate representations for rare and unseen words (Bhatia et al., 2016; Herbelot and Baroni, 2017; Schick and Schutze, 2018) by learning more effective and more informed components of subword-informed configurations. We have presented a general framework for learning subword-informed word representations which has been used to perform a systematic analysis of 60 different subword-aware configurations for 5 typologically diverse languages across 3 diverse tasks. The large space of presented results has allowed us to analyze the main properties of subword-informed representation learning: we have demonstrated that different components of the framework such as segmentation and composition methods, or the use of position embeddings, have to be carefully tuned to yield improved performance across different tasks and languages. We hope that this study will guide the development of new subword-informed word representation architectures. Code is available at: https://github. com/cambridgeltl/sw_study . This work is supported by the ERC Consolidator Grant LEXICAL (no 648909). We thank the reviewers for their insightful comments, and Roi Reichart for many fruitful discussions.
abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain objective abstain abstain abstain abstain abstain abstain method abstain abstain abstain result result objective abstain other abstain abstain abstain abstain abstain abstain method method method abstain method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain abstain abstain method method method method method abstain method method method abstain abstain method method abstain abstain method other abstain abstain method abstain abstain method method method method abstain method method method method method method abstain method method method method other method method abstain abstain abstain abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method objective objective other abstain other other