Lexica distinguishing all morphologically related forms of each lexeme are crucial to many language technologies, yet building them is expensive. We propose Frugal Paradigm Completion , an approach that predicts all related forms in a morphological paradigm from as few manually provided forms as possible. It induces typological information during training which it uses to determine the best sources at test time. We evaluate our language-agnostic approach on 7 diverse languages. Compared to popular alternative approaches, our Frugal Paradigm Completion approach reduces manual labor by 16-63% and is the most robust to typological variation. From syntactic parsing (Seeker and Kuhn, 2013) to text-to-speech (Zen et al., 2016; Wan et al., 2019), many linguistic technologies rely on accurate lexica decorated with morphological information. Yet, building such lexica requires much human effort (Buckwalter, 2002; Tadic and Fulgosi, 2003; Forsberg et al., 2006; Sagot, 2010; Eskan-der et al., 2013). We present a language-agnostic method for minimizing the manual labor required to add new paradigms to an existing lexicon. Formally, let each lexicon entry, or realization, be a triple ( P , C , f ). P marks membership in some paradigm P of morphologically related words, C defines a cell in P as a bundle of morphosyntactic features, and f is the form realizing C in P . Hence, paradigm SING can be expressed (in the UniMorph schema (Kirov et al., 2018)) as a set of realizations: {( SING , NFIN , sing ), ( SING , 3. SG . PRES , sings ), . . . }. sible to be manually realized, e.g., {( FLY , NFIN , fly ), ( FLY , PST , flew )} such that the forms realizing the remaining cells can be predicted, i.e., flies , flying , flown . Here, sources are manually provided realizations. Targets are realizations whose forms must be predicted from sources. Our work differs from traditional paradigm completion (Durrett and DeNero, 2013) in that sources are not given blindly, but the system must strategically select which sources it wants to be given at test time. Paradigm completion from one source is typically non-deterministic due to multiple inflection classes realizing different exponents in some cells, e.g., suffixing + ed generates the past tense for WALK , but not for SING or FLY which are members of different classes. Hence, many works discuss paradigm completion in the context of (implicit) inflection class disambiguation (Ackerman et al., 2009; Montermini and Bonami, 2013; Beniamine et al., 2018). Finkel and Stump (2007) propose three approaches to select the fewest sources required to deterministically identify class. Yet, neural sequence models can often complete paradigms accurately from less sources without fully disambiguating inflection class (Kann and Schtze, 2016; Aharoni and Goldberg, 2017; Wu and Cotterell, 2019). See Elsner et al. (2019) for an overview of the application of neural sequence models to morphological theory. We propose Frugal Paradigm Completion (FPC), inspired by work on inflection class disambiguation and neural sequence modeling. We train a source selection agent (SSA) to induce typological knowledge regarding the distribution of complexity in paradigms and use this to request informative source cells to be realized by an oracle . Sources are fed to a predictor to generate target forms. For each paradigm, SSA iteratively requests sources until the oracle confirms all cells have been realized correctly. We introduce a novel metric, auto-rate , to quantify the manual labour (performed by the oracle) needed to complete each paradigm. Using this metric, we demonstrate that FPC reduces labor by 63% over predicting targets from lemmata, and 47% over predicting them from the smallest set of sources that fully disambiguates inflection class. We propose a new typology for discussing the organization of complexity in paradigms which helps explain why strategies perform better or worse on certain languages while FPC, being sensitive to typological variation, performs robustly. After discussing related paradigm completion approaches in Section 2, we describe FPC in Section 3. Section 4 covers all data and experimental set up details. We discuss results in Section 5 and analyze FPC's behavior in Section 6. Lemma-based Paradigm Completion The standard paradigm completion approach does not select sources, but assumes one source: the lemma (Dreyer and Eisner, 2011), whose distinction is ultimately arbitrary. Yet many have shown that more informative sources can be chosen (Finkel and Stump, 2007; Cotterell et al., 2017b; Kann and Schtze, 2018). Most Informative Source For each target form to be predicted, Kann and Schtze (2018) select the source most likely to predict that form. Unlike FPC, they do not attempt to minimize the number of unique sources that must be manually realized. Static Principal Parts To minimize sources required to fully disambiguate inflection class, Finkel and Stump (2007); Stump and Finkel (2013) propose three approaches: static , dynamic , and adaptive . In the static approach, the same sources must be used for every paradigm (these sources are referred to as principal parts in a much older pedagogical tradition dating back to ancient Rome with Varro's de lingua latina (Grinstead, 1916; Ahern, 1990)). Cotterell et al. (2017b) train a model on static sources and attain near 100% accuracy in Latin verb paradigm completion. However, they do not consider that one paradigm may require fewer sources than another, nor that paradigm completion may require fewer sources than inflection class disambiguation. Dynamic Principal Parts Finkel and Stump (2007)'s dynamic approach selects a minimal set of sources necessary to fully disambiguate inflection class which can be unique to that inflection class. While efficient, this is impractical in that it requires oracular knowledge of class prior to seeing any forms. Adaptive Principal Parts Finkel and Stump (2007)'s adaptive approach, like our FPC method, chooses the same first source cell for each paradigm P . Subsequent sources are selected conditional on the set of inflection classes P could belong to given the sources realized so far. Hence, the number of sources required per paradigm is upper bounded by the static approach and lower bounded by the dynamic. Our FPC approach is a neural update, inspired by their adaptive approach. While their implementation tracks viable inflection classes explicitly with rules operating on oracularly segmented af-fixes, we use sequence models operating on whole words to remove reliance on oracular segmentation and leverage stem-internal phonology known to correlate with inflection class (Aronoff, 1992; Dressler and Thornton, 1996; Dawdy-Hesterberg and Pierrehumbert, 2014). This section describes the interactions of the three FPC components. As illustrated in Figure 1, the predictor takes a source cell and its realizing form as input, e.g., 3. SG . PRES : sings , or cell 2 : form 2 in the figure. The predictor is composed of as many sub-predictors as there are cells in the paradigm, each of which is trained to predict the entire paradigm from one source cell's realization. Cell 2 in the paradigm is grayed out in the figure, as this was provided as input so it does not have to be predicted. The predicted paradigm is evaluated by the oracle. If there are no errors, we are done. Otherwise, based on previous sources, SSA chooses a new cell to be realized by the oracle and gives it to the predictor as the next source. Because cell 3 is chosen in the figure, sub-predictor 3 will be used to predict the paradigm going forward, and cells 2 and 3 will both be grayed out. The process continues like this until all cells have been correctly predicted by at least one sub-predictor. Crucially, during inference, each test paradigm is empty, i.e., no realization has been seen during training and no source is available to inflect from  sub-predictor 1 sub-predictor n ... sub-predictor 2 predictor cell 1: form 1 cell n : form n cell 2: form 2 paradigm ... oracle done error source selection agent (SSA) cell3: form 3 cell 2: form 2 input cell 3 oracle Figure 1: Schematic representation of the flow of Frugal Paradigm Completion at inference time. a-priori . Our setup aims to minimize the number of sources which the SSA must request from the oracle (typically a human in the loop at inference time) to predict the remaining paradigm slots correctly. The predictor outputs a target form given its cell, a source form and the source form's cell as input. To train the predictor, for each possible source cell, we train a sub-predictor to predict every possible target form in every paradigm in the training data given the realization of that source cell in that paradigm. Details of all sequence model architectures are provided in Section 4. 3.2 Source Selection Agent SSA's choice of a cell for a given paradigm depends on all previously selected cells for that paradigm and their corresponding forms. This allows SSA to learn, e.g., that given a previous English PST source, PST . PTCP should only be requested as a subsequent source if the PST form did not take the regular ed suffix. Otherwise, PST . PTCP is likely to be regular and unlikely to contribute new information. To induce such knowledge, we train SSA on an oracle policy of ideal source selections extracted from the train set (Ross et al., 2011; Ross and Bag-nell, 2014; Welleck et al., 2019). 1 To extract the oracle policy, we divide the training lexicon into two folds and train one predictor on each, allowing us to cross-validate each predictor on its held out fold. For each training paradigm, we test which target forms can be correctly predicted by which source cells' sub-predictors. As shown for SING 1 While we borrow the term oracle policy from Imitation Learning (Ross et al., 2011; Ross and Bagnell, 2014; Welleck et al., 2019), we mimic the oracle policy with simple sequence learning. Our analysis suggests even this may be more machinery than necessary. in Figure 2, we use this information to extract minimum set covers, i.e., the fewest source cells such that the union of the subsets they predict correctly covers the entire paradigm. These covers constitute the oracle policy used to train SSA. The minimum set cover problem is NP-complete (Lund and Yannakakis, 1994; Kuhn et al., 2005), but we approximate it in O ( log e | P | ) by iteratively selecting the cell whose subset most enlarges the union. We break ties by averaging predictiveness (Equation 1) over both folds, where fold F contains | F | paradigms; P m , | P m | cells; and Acc ( P m , C trg , C src ) returns 1 if using C src 's realization as a source correctly predicts the form realizing cell C trg in paradigm P m . predictiveness ( C src , F ) = (cid:80) | F | m =1 (cid:80) | P m | j =1 Acc ( P m , C j , C src ) (cid:80) | F | m =1 | P m | (1) At this stage, paradigm covers are dynamic in that no single cell need be shared by all covers. Yet, when selecting the first source, SSA has no previous sources to condition on, making it impossible to predict the first cell. Thus, we get adaptive minimum set covers by designating the start cell to be that which occurs in the most dynamic covers. Then we re-approximate all covers such that each includes this cell. 2 Finally, we rank cells within each cover by the total number of covers in which they appear. For each cell in each cover, we train SSA to predict said cell from all higher ranked cells and their realizing forms (holding out 2% of them for development). 2 We train and test on a single part-of-speech for each language, so each paradigm should contain the start cell. For defective paradigms lacking said cell, we back off to the most frequent cell that exists in the paradigm. The oracle represents a human-in-the-loop during inference, providing requested source realizations to the predictor and informing SSA when a paradigm is complete and accurate (Figure 1). In our implementation, the oracle does not specify which individual predictions are incorrect, but it thus must resolve any discrepancies when two sub-predictors disagree after the fact. We do not attempt to model the additional cost this incurs, as it is unclear how to combine it with the presumably more expensive cost of correcting errors, which we model instead. This is worth re-visiting in future work. We evaluate 4 paradigm completion approaches on 7 languages. Here we discuss implementation, data and evaluation details. All sequence models in all implementations of any paradigm completion approach use the Transformer architecture (Vaswani et al., 2017). Here we describe the formatting of input and outputs as well as our hyperparameters. Input and Output Formats Following Kann and Schtze (2016), input sequences combine characters and morphosyntactic features. The following is a sample input and output for a single source FPC sub-predictor specializing in the cell NFIN : Input: f l y out_ V . For any inflected-form-predicting sequence model whose input is not limited to realizations of a single cellas in, e.g., the static principal parts approachsource cell features are prepended to the input as such: For multi-source sequence models, the features of each source are inserted into the input and the target features are listed after the first source. We experimented with several different multi-source representations and the Transformer performed fairly similarly with all of them. Input: in_ NFIN f l y out_ V . PTCP out_ PST in_ PST f l e w Output: f l o w n The FPC's SSA predicts not a form, but a cell, conditional on any previously realized sources. To predict the first source, it is given nothing and will thus deterministically select the best starting cell as determined by the oracle policy (see Section 3.2). To predict any subsequent source, it conditions on the realizations of all previously requested sources for that paradigm. The following exempli-fies SSA inputs and outputs when predicting the second source for paradigm FLY : Input: in_ NFIN f l y Output: in_ V . Wu et al. (2018) and others have achieved improvements by embedding morphosyntactic features separately and concatenating them to the encoder output prior to feeding it to the decoder. Our error analysis, however, suggests Transformers handle Kann and Schtze (2016)-style input well. More sophisticated feature handling may not be necessary, but should be investigated in future work. Hyperparameters We train all Transformer models for 100 epochs in batches of 64 with 0.1 dropout probability. The final model is restored from the epoch with the highest dev accuracy. We stop early if there is no improvement for 20 Train Dev Test Arabic nouns paradigms 1006 100 100 instances 24160 2260 2352 German verbs paradigms 1031 100 100 instances 27762 2690 2692 English verbs paradigms 2908 200 201 instances 14522 1000 1001 Russian nouns paradigms 3289 100 100 instances 37423 1133 1137 Latin nouns paradigms 1630 100 100 instances 19150 1180 1174 Hungarian nouns paradigms 1405 100 100 instances 47689 3400 3383 Irish nouns paradigms 549 100 100 instances 6460 1197 1195 Table 1: Number of paradigms and instances by split for every language and POS considered. epochs. The only exception is during FPC cross-validation where sub-predictor models are trained for only 50 epochs with early stopping after 5 epochs without improvement. This is just to reduce computational cost as it is sufficient to induce an oracle policy. The final sub-predictor models however (those used at inference time, not those used to induce the oracle policy), are trained on the full training data set using the full 100 epochs with 20 epochs patience for early stopping. As for Transformer-specific hyperparameters, using the original notation of Vaswani et al. (2017), we set N = 4 , d model = 128 , d ff = 512 , and h = 8 , scaling down the hyperparameters recommended for machine translation as our task is less expensive (Aharoni and Goldberg, 2017; Wu et al., 2018). For every language and part of speech (POS) considered, we extract train, dev and test sets from UniMorph (Kirov et al., 2018). Each split contains full paradigms, though the cells realized in each may vary due to defectiveness (Corbett, 2005; Sims, 2015). We filter many gold errors by removing paradigms for which no realization can be attested in actual text. We use Universal Dependencies (UD) (Nivre et al., 2016) to check for attestations. We also filter overabundant realizations (multiple forms realizing one cell), keeping only the most frequent form, as attested in UD. While some languages allow for overabundance (Thornton, 2010, 2011), in UniMorph, this often indicates a gold error. We randomly divide paradigms into splits such that train is maximally large and dev and test contain at least 100 paradigms and 1,000 realizations. Exact quantities are displayed in Table 1. Arabic, German, English, and Russian were used for development, while Irish, Hungarian, and Latin were only evaluated after fixing hyperparameters. The languages considered represent 3 families and 4 diverse Indo-European branches. They exhibit multiple non-canonical behaviors (Corbett, 2005) and present diverse challenges from non-concatenative morphology to complex inflection class systems. Paradigm completion is usually evaluated via exact match accuracy on held out target forms (Cot-terell et al., 2016, 2017a, 2018; McCarthy et al., 2019). Yet we use as many sources as are necessary to reach 100% accuracy in predicting the remaining slots, so accuracy is not a meaningful metric for the FPC. Some theoretical works focus on the sources required to unambiguously complete a paradigm given some implicit knowledge of viable inflection classes (Finkel and Stump, 2007; Ackerman and Malouf, 2013). Yet these tend not to propose actual paradigm completion models or evaluate their decisions in ambiguous cases. To evaluate our system and bridge these traditions, we propose auto-rate : auto-rate = (cid:80) ni =1 auto ( P i ) (cid:80) ni =1 | P i | , (2) where auto ( P ) denotes the number of realizations correctly predicted while not having been provided as sources for paradigm P by the oracle. Intuitively, auto-rate is like accuracy but it counts oracularly provided sources as additional errors since both errors and sources require labor, i.e., sources require manual input and errors, post-correction. We also report manual cells per paradigm , i.e., sources plus errors. Of course, FPC resolves all errors eventually, but other systems can make errors requiring post-correction. We compare the FPC method to three baselines. One is a variant of FPC using a random SSA. This allows us to distinguish the benefit of a smart SSA from that of simply receiving additional feedback from an oracle in the loop. Each time a source must be selected, random SSA chooses randomly without replacement. Its performance is averaged over two runs. The lemma approach baseline predicts all paradigm forms from one designated source: the lemma. Finally, for the static approach baseline, we considered two static approach implementations. The single-source implementation predicts each target from the source that is, in theory, its best predictor (Kann and Schtze, 2018). The multi-source implementation concatenates these sources, predicting each target from the concatenated input. As results are nearly identical for either implementation, we report results only for single-sourcewith the exception of Latin, as explained presently. For some languages, there is little theoretical or pedagogical literature to help identify the best sources for the static approach. Our single-source static approach for Arabic nouns predicts singular and dual forms from SG ; NDEF ; NOM and plurals from PL ; NDEF ; NOM . In theory, any non-plural plus any plural should be sufficient (Brustad et al., 2005; Habash, 2010). For German verbs, we predict present and imperative forms from NFIN and past forms from IND ; PST ;1 SG (Grebe et al., 1966). We predict English present forms from NFIN ; PST and V . PTCP ; PST predict themselves. For Russian nouns, Zaliznyak (1980) argues for five sources, yet Parker (2016) demonstrates that three are usually sufficient. We follow the latter, predicting all nominative or accusative forms from ACC ; SG , all other singulars from INS ; SG , and all other plurals from GEN ; PL . In preliminary experiments, we found this to match the accuracy of the five source approach, thus achieving a higher auto-rate. For Latin, we could not evaluate a single-source static implementation as it is unclear which source cell best predicts each target. The multi-source static approach for Latin nouns predicts all forms from NOM ; SG and GEN ; SG (following the classical grammatical analyses of Varro, Priscian and the Roman ars grammatica ). For Irish and Hungarian, we do not evaluate a static approach as we lack the requisite linguistic knowledge to determine the best sources. As shown in Table 2, FPC always ties or beats the next best approach, while the next best approach varies by language. On average, FPC reduces labor by 63% over the lemma approach, 47% over static, 16% over random agent, and 13% over the next best approach. Its success is mainly due to (1) making predictions from fewer sources than are required for fully disambiguating inflection class and (2) receiving feedback after each source. Surprisingly, training a sophisticated SSA does not improve much over using a random agent. We argue this is due to an unexpectedly large margin of error in the agent's source selection task. Despite the complexity of source selection strategies required for inflection class disambiguation, FPC uses lexical frequencies to expect regularity and stem-internal clues to anticipate irregular classes, requiring a median of just one source per paradigm for all languages except under-resourced Irish. Furthermore, inspection of the source selection minimum set covers reveals that it is often the case that a paradigm can be completed correctly from any single source. This is surprising in light of the precise strategies required for completely deterministic paradigm completion in Finkel and Stump (2007)'s framework and in light of Albright (2002)'s case for the privileged status of a single form per paradigm, though in our framework with full words and full paradigms for training, it seems that many sources can often serve as good enough singleton principal parts. This supports Bonami and Beniamine (2016) proposal of gradient principal part analyses. Here, we discuss patterns relating SSA's first and second sources chosen (Figures 3a-b and 4a-b) to the inter-predictability of cells represented by heat maps (3c and 4c). Maps display the average accuracies with which each target (column) can be predicted from each source (row). We analyze spe-cific SSA choices and predictor errors in Arabic and Latin. The maps (for all languages, see the Appendix) suggest complexity can be distributed within paradigms in systematically distinct ways. Ackerman and Malouf (2013) propose integrative (I-) complexity, using average conditional entropy to describe paradigmatic organization, but this has been criticized for obscuring differences in the predictability of sub-paradigm regions (Cot-terell et al., 2019; Elsner et al., 2019). To remedy this, we propose a typology for measuring the extent to which I-complexity is realized via different organizational strategies, which is useful for discussing source selection strategies. Our typology describes paradigms in terms of mutual predictability , the correlation of a map and its transpose, and entropy predictiveness , the negative correlation of cells' average predictiveness (see Equation 1) and average predictability, defined here in comparable terms as: predictability ( C trg , F ) = (cid:80) | F | m =1 (cid:80) | P m | j =1 Acc ( P m , C trg , C j ) (cid:80) | F | m =1 | P m | (3) Intuitively, a paradigm is mutually predictable if the fact that cell A predicts cell B means that B is likely to predict A . Such paradigms often feature regions of mutually predictable cells (as in 3c), such that an optimal strategy avoids picking multiple sources from one region. For entropy predictive paradigms, if A is generally more difficult to predict than B , A is likely to be a better predictor of the remaining cells (following the information theoretic logic that surprisal is informative (Shannon, 1948; Jaeger, 2010)). For such paradigms, the optimal strategy selects the source which would have been the most difficult target to predict. Unlike Sims (2020)'s graph-theoretic typology for describing inflection class structure, our typology is a two-dimensional description of how the optimal paradigm completion strategy is affected by underlying class structure. In this sense, our typology is complementary to hers and future work might investigate the relationship between traits in her typology and mutual predictability or entropy predictiveness. Furthermore, our typology might be updated to consider the impact of type frequency (Sims and Parker, 2016) in a framework where distributional data is available. Figure 5 demonstrates that cross-linguistic variation is vast with respect to our typology, as some languages even exhibit negative entropy predictiveness or mutual predictability. This partly explains why non-FPC approaches perform erratically: if paradigmatic organization varies by language, source selection strategies must be able to adapt to the data. Arabic nouns are mutually predictable (Figure 5). Any singular or dual form can predict another. Plural forms also predict each other. Yet, in general, plurals are less predictive/able (Figure 3c) due to several inflection classes varying in the plural. The sound plurals take suffixes while broken plural classes are realized via non-concatenative processes. For example, (cid:73)(cid:46)(cid:187) (cid:64)(cid:80) rAkb , rider from root (cid:72) (cid:46) (cid:188) (cid:80) r k b , takes the broken plural pattern _ _ A _ , becoming (cid:72) (cid:46) (cid:65)(cid:191)(cid:80) rkAb . Yet, having heard only singular realizations, a human might posit a sound plural, i.e.,  (cid:9)(cid:224)(cid:241)(cid:74) (cid:46) (cid:187) (cid:64)(cid:80) rAkbwn , realizing the SSA learns an ideal strategy, requesting a singular source (Figure 3a) and then a plural (3b). Interestingly, 6 of 18 sound feminine plurals (most frequent single class) require multiple sources and 8 of 28 broken plurals do not. Thus, the predictor does not default to regularity, but uses stem-internal phonology to anticipate irregularity. Most errors made from the first source posit a viable broken plural, just not the right one. In future work, modeling semantics can fix such errors, e.g., knowing that (cid:73) (cid:46) (cid:187) (cid:64)(cid:80) rAkb is animate makes plural For future work, we can pre-train on raw corpora to give our model access to such information (Devlin et al., 2019). Indeed Erdmann and Habash (2018) found distributional information to benefit inflectional paradigm clustering in Arabic. Though the benefits should generalize as semantics correlates with inflection class in many languages (Wurzel, 1989; Aronoff, 1992; Harris, 1992; Noyer, 1992; Carstairs-McCarthy, 1994; Corbett and Fraser, 2000; Kastner, 2019). Latin is not mutually predictable with moderate entropy predictiveness. SSA's choices are, at first, opaque, but Table 3 shows that ACC ; PL nar-rows the inflection class to variants of one declension. Remaining ambiguity mostly involves 3 rd declension nominative and vocative realizations, which can usually be predicted from the preferred second source cell, VOC ; SG . 44 of 100 test paradigms were 3 rd declension, which required multiple sources at the highest rate (16 of 44; 2 nd masculine declension was next highest at 3 of 15). There was no correlation between declension and second source chosen, yet high auto-rate suggests SSA's choices may not need to condition on previously realized source forms, but only their cells. from a single source, we found paradigms requiring three sources that might be completable from two using a multi-source FPC implementation. For example, greges , flocks realizes GREX . ACC ; PL , but the predictor mistakenly posits  gregium for GEN ; PL from this source, guessing the wrong 3 rd declension variant. While second source VOC ; SG grex corrects this, it obscures the underlying stem, as x can be an allophone of g or c . Thus, we still get an error,  grecum . A multi-source predictor could avoid forgetting the underlying allophone g after seeing the second source. 3 That said, multi-source FPC is not as simple as multi-source static. Heuristic sampling of training instances based on the oracle policy yields predictors that only attend to one source or make bad predictions when only given one. This is worth exploring further in future work as there is more evidence of paradigms that are difficult to handle without jointly encoding sources in the linguistic literature (Corbett, 2005; Bonami and Beniamine, 2016). We presented Frugal Paradigm Completion, which reduces the manual labor required to expand a morphological lexicon by 16-63% over competitive approaches across 7 languages. We demonstrated that typologically distinct morphological systems require unique treatment and benefit from our SSA, that learns its strategy from data. We found that inducing this strategy is not as challenging as previously suggested (Finkel and Stump, 2007). Thus, SSA might be replaced with a less costly architecture while our model might be improved by conditioning on semantics and jointly decoding from a variable number of sources. We are indebted to helpful conversations with Shijie Wu, Ryan Cotterell, Katharina Kann, Andrea Sims, and Olivier Bonami. We would also like to acknowledge insightful feedback from Google's Word Graph and TTS teams, as well as four anonymous reviewers. 3 See rexregis , king or paxpacis , peace , which are technically conditioned on preceding vowel quality, though there are probably not enough training examples for the model to learn that.
abstain objective abstain method method abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method abstain abstain objective objective objective abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain method abstain abstain abstain abstain abstain abstain method method method abstain abstain method abstain method method abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain other abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method objective result result other other other