Recent parameter-efficient language model tuning (PELT) methods manage to match the performance of fine-tuning with much fewer trainable parameters and perform especially well when training data is limited. However, different PELT methods may perform rather differently on the same task, making it nontrivial to select the most appropriate method for a specific task, especially considering the fast-growing number of new PELT methods and tasks. In light of model diversity and the difficulty of model selection, we propose a unified framework, UNIPELT, which incorporates different PELT methods as submodules and learns to activate the ones that best suit the current data or task setup via gating mechanism. On the GLUE benchmark, UNIPELT consistently achieves 1~4% gains compared to the best individual PELT method that it incorporates and outperforms fine-tuning under different setups. Moreover, UNIPELT generally surpasses the upper bound that takes the best performance of all its submodules used individually on each task, indicating that a mixture of multiple PELT methods may be inherently more effective than single methods. 1 1 Introduction As pre-trained language models (PLMs) (Devlin et al., 2019) grow larger and larger (Brown et al., 2020), it becomes increasingly infeasible to perform conventional fine-tuning, where separate replicas of the model parameters are modified per single task. To solve the issue, there has recently been a surge of studies on p arametere fficient l anguage model t uning (PELT), namely how to effectively tune the PLMs with fewer trainable parameters. Existing PELT research generally aims at achieving performance comparable to fine-tuning with  Work was done during internship at Meta AI. as few trainable parameters as possible, which has seen significant progress  the task-specific trainable parameters used in most recent approaches (Lester et al., 2021; Guo et al., 2021) are almost negligible compared to the total parameters of the PLM (<1%). A more challenging yet less studied problem is whether one can achieve better performance than fine-tuning with fewer parameters. Recent studies (He et al., 2021; Li and Liang, 2021; Karimi Mahabadi et al., 2021b) find that some PELT methods are more effective than fine-tuning on certain tasks when training data is limited, possibly due to the reduced risk of overfitting. However, as found in our experiments (Table 1), different PELT methods exhibit diverse characteristics and perform rather differently on the same task, which 6253 makes it nontrivial to select the most appropriate method for a specific task, especially considering the fast-growing number of new PELT methods and tasks (Ding and Hu, 2021). In light of the diverse performance of PELT methods and the cost of selecting the best method, we propose a unified PELT framework, named UNIPELT, which incorporates different PELT methods as submodules and learns to dynamically activate the (combination of) submodules that best suit the current data or task setup. As a result, model selection is no longer needed and consistently better performance is achieved under different setups. The activation of each submodule in UNIPELT is controlled by gating mechanism , which learns to favor (assign more weight to) the submodules that positively contribute to a given task. In addition, since the number of parameters introduced by each submodule is generally small, combining multiple methods leads to negligible losses in model efficiency. We select four representative PELT methods for our study  adapter (Houlsby et al., 2019), prefix-tuning (Li and Liang, 2021), LoRA (Hu et al., 2021), and BitFit (Ben Zaken et al., 2021), which largely cover the major categories of PELT methods. We perform two sets of analysis that carefully examines (i) the characteristics of individual PELT methods and (ii) their effectiveness when coordinated by UNIPELT under various setups. 2 Extensive experiments on the GLUE benchmark (Wang et al., 2019), with 32 setups (8 tasks  4 data sizes) and 1,000+ runs, not only reveal the diverse behavior of PELT methods, but also show that UNIPELT is more effective and robust than using each method alone in various task and data setups. Specifically, UNIPELT consistently improves the best submodule that it incorporates by 1~4 points and even outperforms fine-tuning, achieving the best average performance on the GLUE benchmark under different setups. Moreover, UNIPELT generally surpasses the upper bound that takes the best performance of all its submodules used individually on each task, which suggests that UNIPELT maintains (near) optimal performance under different setups. The fact that UNIPELT outperforms the upper bound also implies that a mixture of PELT methods involving different parts of the PLM architecture may be inherently more effective than 2 BitFit is not included in UNIPELT as it typically performs the worst in our preliminary experiments. Contributions . (1) We conduct a comprehensive study of representative PELT methods and carefully examine their differences and commonalities in terms of performance and characteristics. (2) We propose a unified PELT framework that can incorporate existing methods as submodules and automatically learn to activate the appropriate submodules for a given task. (3) Our proposed framework achieves better average performance than fine-tuning and the PELT methods that it incorporates under various setups, often performing the best and never the worst at per-task level, exhibiting supe-rior effectiveness and robustness with negligible losses in model efficiency. PLMs can be used as feature extractors where only the top layers or prediction head are fine-tuned without additional parameters (Lee et al., 2019). However, such fine-tuning approaches generally lead to degenerate model performance that is much worse than fine-tuning all parameters (Lee et al., 2019; Pfeiffer et al., 2021). A recent method BitFit (Ben Zaken et al., 2021) only tunes the bias terms of the PLM and is shown to achieve performance comparable to fine-tuning on certain tasks when training data is limited. Therefore, we select BitFit as the representative of this category for analysis. Alternatively, one may fix the entire PLM and introduce a small number of new trainable parameters. Notable examples in this category include adapter (Houlsby et al., 2019) and its extensions (Pfeif-fer et al., 2021; Karimi Mahabadi et al., 2021b), prefix-tuning (Li and Liang, 2021) and its extensions (Lester et al., 2021), and additive methods (Guo et al., 2021; Hu et al., 2021). Next, we will briefly describe these methods to facilitate the introduction of our proposed framework. An illustration is shown in Fig. 1 for better understanding. Adapter . Adapter (Houlsby et al., 2019) adds a trainable bottleneck layer after the feedforward network in each Transformer layer of the PLM. A bottleneck layer consists of a down+up projection pair that shrinks and recovers the size of token hidden 6254 states. Mathematically, if we denote the output of the f eedforward network after residual connection and layer n ormalization as h FN with hidden size D hidden and bottleneck size D mid , then the output of a bottleneck layer h A is: h A = W (cid:124) up  ( W (cid:124) down h FN ) , (1) where W down  RD hidden  D mid , W up  RD mid  D hidden ,  is a nonlinear activation function, and the bias terms are omitted for brevity. The parameters in layer normalization and the final prediction head sometimes are also fine-tuned depending on the specific adapter variants. Adapter has shown to be on par with fine-tuning and sometimes exhibits better effectiveness in the low-resource setting (He et al., 2021). Later studies extend adapter to multi-lingual (Pfeiffer et al., 2020b) and multi-task (Karimi Mahabadi et al., 2021b) settings, or further reduce its trainable parameters (Karimi Mahabadi et al., 2021a), which can be easily incorporated into UNIPELT as a replacement of the vanilla adapter. Prefix-tuning . Prefix-tuning (Li and Liang, 2021) prepends a number of task-specific trainable vectors to the input of multi-head attention in each Transformer layer, which the original tokens can attend to as if they were virtual tokens. Specifically, we denote the original sequence length L 0 , the number of trainable vectors ( i.e. , prefix length) L , and the Transformer layer input h in  RD hidden  L 0 . First, three linear projections WQ , WK , WV  RD hidden  D hidden transform h in into Query Q , Key K , and Value V . Then, two prefix matrices PK and PV  RD hidden  L are prepended to K and V . To stabilize optimization, the prefix matrix P is reparameterized by a feedforward network: P (cid:48) = W (cid:124) up  ( W (cid:124) down P ) , (2) where W down  RD hidden  D mid , W up  RD mid  2 N layer D hidden , and N layer denotes the number of Transformer layers. The parameters of this network can be discarded after training, and only 2 N layer prefix matrices  RD hidden  L are needed (2 matrices for each layer). Prefix-tuning is originally evaluated on natural language generation and we adapt it to understanding tasks. A follow-up method named prompt-tuning (Lester et al., 2021) further reduces task-specific parameters by limiting the prefix to the first layer but only performs competitively with very large model sizes (billions of total parame-ters), and is thus not considered in our study. Note that prefix-tuning (or prompt-tuning) is different from prompt-based fine-tuning methods (Schick and Schtze, 2021; Gao et al., 2021) (see App. A for specific differences). Additive Methods . Additive PELT methods treat the model parameters after fine-tuning as an addition of the pre-trained parameters  pre-trained and task-specific differences  task , where  pre-trained is fixed and a new (sub)set of model parameters are added on top:  task =  pre-trained +  task . There are various ways to parameterize  task , leading to different additive methods such as LoRA (Hu et al., 2021), diff pruning (Guo et al., 2021), and side-tuning (Zhang et al., 2020). We take LoRA as a representative and incorporate it into UNIPELT. Other methods are conceptually similar and can be incorporated in the same fashion. LoRA introduces trainable low-rank matrices and combines them with the original matrices in the multi-head attention. Specifically, two matrices W down  RD hidden  D mid and W up  RD mid  D hidden are added for the query and key projections along with the original matrix WQ and WK  RD hidden  D hidden : Q = ( W (cid:124) Q +  W (cid:124) up W (cid:124) down ) h in , (3) where  is a fixed scalar hyperparameter for scaling the task-specific differences. The form of the trainable matrices in LoRA is quite similar to those in adapter or prefix-tuning, but there is no activation function  in between. Given a large PLM M with size |M| that cannot be fine-tuned directly due to computational or storage cost, suppose that we have a list of PELT methods { m i } , the trainable parameters of which are negligible ( i.e. , (cid:80) i | m i | (cid:28) |M| ), our goal is to design a unified PELT framework that incorporates { m i } as submodules and learns to dynamically activate (up-weight) different submodules when appropriate under different scenarios, such that one could achieve satisfactory results in terms of both model effectiveness and robustness without the hassle of permuting all the method  task  data combinations. Motivation & Intuition . During the analysis of individual PELT methods, we observe that different PELT methods exhibit diverse characteristics and perform rather differently on the same task. For example, prefix-tuning generally performs well on natural language inference tasks regardless of the size of training data. Also, as can be seen in Fig. 1 and Sec. 2, different PELT methods often involve different parts of the PLM architecture ( e.g. , before multi-head attention for prefix-tuning and after feedforward layer for adapter), making it feasible to combine multiple PELT methods without (directly) interfering with each other. In light of the two observations above, we propose a unified PELT framework, UNIPELT, which takes a hybrid approach by incorporating multiple PELT methods as submodules. At a high level, UNIPELT improves over single PELT methods due to two factors. First, UNIPELT learns to activate (upweight) the submodules that best suit the current task or specific data sample and deactivate (down-weight) the rest. Second, we find that UNIPELT generally performs better than taking the best performance of all its submodules used individually on each task, suggesting that there could be some compounding effects that lead to better model effectiveness when multiple PELT methods (that modify different parts of the PLM) are used. Next, we will introduce how different PELT methods can be incorporated into UNIPELT via gating mechanism. Gating Mechanism . To achieve fine-grained control of submodule (de)activation, we add a trainable gate G m i for each submodule m i  { A, P, L } in every Transformer layer (see Fig. 1). The letters A, P, L stand for Adapter, Prefix-tuning, and LoRA, respectively. Intuitively, if m i is useful for a given data  task setup (or a particular instance), the gate output for m i would be higher such that m i plays a more important role. The actual interplay of submodules, however, is more complicated given the interdependency of the submodules and the compounding effects of multiple layers. Specifically, for adapter, there is a residual connection between the feedforward network and the adapter submodule that sums the adapter input (be-fore normalization) h F and output h A as its final output: h (cid:48) A = h A + h F . We design a gating function GA  (0 , 1) that estimates the importance of adapter by its direct input h FN using a feedforward network with sigmoid activation and then scales its output: h (cid:48) A = GA h A + h F . The adapter submodule is effectively bypassed if GA  0 . Similarly, for prefix-tuning, we design a gating function GP  (0 , 1) that is applied to the prefix vectors ( PK and PV ) with the representation of the original tokens ( K and V ) intact. In this way, the impact of the prefix would be diminished if the gate output of the prefix-tuning submodule is low. 3 The gating function GP is estimated by the Transformer layer input h in with another feedforward network. As for LoRA, we note that there is already a constant scaling factor  in its original design that resembles the purpose of our gating mechanism. We thus simply make the factor learnable per layer by a third feedforward network that takes h in as input instead of specifying a constant manually:  task =  pre-trained + GL  task . Despite the seeming simplicity of UNIPELT, we note that it is nontrivial for a unified approach to work well under different scenarios. Naively combining different PELT methods as a hybrid approach could lead to mixed or worse performance than using individual methods, as observed in both our experiments and prior studies (Hu et al., 2021). We conduct extensive experiments with 8 tasks  4 data sizes  7 methods  5 runs per setup, along with additional analysis for particular methods, resulting in 1,000+ runs in total. Task Setup . We conduct experiments on the General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2019), which involves four types of natural language understanding tasks including linguistic acceptability (CoLA), sentiment analysis (SST-2), similarity and paraphrase tasks (MRPC, STS-B, QQP), and natural language inference (MNLI, QNLI, RTE). We exclude the WNLI dataset following prior studies (Houlsby et al., 2019; Devlin et al., 2019). Data Setup . We mainly consider a low-resource setting where training data is limited and the performance of different methods varies much. We sample a small subset of the training set for each task with size K = { 100 , 500 , 1000 } . As it is infeasible to submit considerable runs to the GLUE 3 Prefix-tuning cannot be fully eliminated as adapter or LoRA due to the softmax operation in multi-head attention. leaderboard (2 submissions/day), we take 1,000 samples on the training set as the development set to select the best checkpoint and use the original development set as the test set. To reduce variance, we shuffle the data with 5 random seeds and report the average performance. Additionally, we consider a high-resource setting where the whole training set is used and the best performance on the GLUE development set is reported. Compared Methods . We mainly compare UNIPELT with fine-tuning and four representative PELT methods: adapter (Houlsby et al., 2019), prefix-tuning (Li and Liang, 2021), BitFit (Ben Zaken et al., 2021), and LoRA (Hu et al., 2021). For completeness, we consider two model variants UNIPELT (AP) and UNIPELT (APL), which incorporate 2 and 3 PELT methods, respectively. Implementation Details . We use BERT base (De-vlin et al., 2019) as the base model in the experiments. Consistent results are observed in our preliminary experiments with BART large (Lewis et al., 2020) (provided in App. C). We implement and evaluate all the methods in the same codebase to ensure a fair comparison. We largely follow the default hyperparameters of different methods and keep them the same on all the tasks for generalizability. We set the prefix length L = 10 , adapter bottleneck size D mid = 48 , LoRA rank D mid = 8 if not specified otherwise. 4 More implementation and hyperparameter details can be found in App. B. 4.2 Analysis of Individual PELT Methods In Table 1, we show the performance of different methods on the GLUE benchmark with various sizes of training data. The results on the development sets are generally consistent with the test sets and provided in App. D. Although the average performance of different methods over 8 tasks is sometimes similar, the differences between tasks are quite significant under certain setups and can be as large as 5~9 points on a specific task ( e.g. , STS-B and MNLI, K = 500 ) even when excluding cases where some methods fail to learn effectively ( e.g. , prefix-tuning on QQP, K = 100 ). 4 While these hyperparameters may lead to differences in trainable parameters, we keep them for analysis as they are used by the official implementation. Also, we observe that more trainable parameters do not guarantee better results. individual PELT method more closely. Analysis of Adapter . The performance of adapter is relatively stable  there is no significantly better or worse result than fine-tuning consistent on different tasks or sizes of training data. In general, adapter is slightly worse than fine-tuning in most cases. We do not observe that adapter consistently outperforms fine-tuning in the low-resource setting as in He et al. (2021), possibly because they tune model hyperparameters on each task, which could be computationally prohibitive when there are considerable tasks. For example, they choose the bottleneck size D mid from {64, 128, 256}, while D mid = 48 is fixed across tasks in our experiments. Also, we only add one adapter in each Transformer layer instead of two following Pfeiffer et al. (2021). These two differences result in 62.4%~90.5% fewer parameters than the adapter used in He et al. (2021). To further study the effect of bottleneck size D mid in adapter, we increase D mid and re-evaluate adapter on a setup that it performs poorly (CoLA, K = 100 ). As shown in Fig. 2, the performance of adapter is increased gradually and becomes significantly better only when D mid = 256 , which involves 5.3  trainable parameters than the adapter used originally ( D mid = 48 ), 4.3  than UNIPELT (AP), and 3.4  than UNIPELT (APL), suggesting that a larger bottleneck size could be beneficial when adapter learns ineffectively. On the other hand, there are certain tasks ( e.g. , STS-B) that adapter largely outperforms competitive methods such as prefix-tuning and LoRA regardless of the size of training data, suggesting that one should favor adapter over other PELT methods under certain scenarios as well. poorly with K = { 100 , 500 } and becomes on par with fine-tuning when K reaches 1000. We also observe that prefix-tuning fails to learn effectively on certain tasks when the training data is limited ( e.g. , K = 100 on SST-2 and K = 500 on QQP), leading to unsatisfactory performance and (or) large variance across different runs. Similar phenomena have been observed in a concurrent study (Gu et al., 2021) on few-shot prompt-tuning. To ensure that the poor performance of prefix-tuning is not due to its fewer trainable parameters (based on its default setting), we further increase the prefix length to L = 50 such that its trainable parameters are comparable to adapter, and reevaluate prefix-tuning on all 8 tasks with K = 100 . For the 4 tasks where prefix-tuning ( L = 10 ) performs poorly (SST2, CoLA, STS-B, and QQP), while its performance is significantly improved on 3 tasks, it also performs significantly worse on the other task (STS-B), which suggests that training instability in the low-resource regime is still an issue for prefix-tuning even with more trainable parameters. 5 Besides, prefix-tuning ( L = 50 ) still lags behind adapter or UNIPELT (AP) on 3 of the 4 tasks. Furthermore, the average performance of prefix-tuning ( L = 50 ) on 8 tasks is even slightly worse than with L = 10 , which indicates that increasing prefix length may not be a panacea for all the scenarios. A larger L also leads to significant training/inference slowdown due to the costly multi-head attention. More broadly, such results suggest that using more trainable parameters does not guarantee better performance. On the bright side, prefix-tuning performs well on certain tasks such as natural language inference (RTE and MNLI) with various sizes of training data, which suggests that one should also prefer prefix-tuning in certain cases. Analysis of BitFit & LoRA . Tuning only the bias terms of the model does not lead to very satisfactory results in our experiments  BitFit never performs the best and generally performs the worst in different data and task setups. Therefore, we do not consider BitFit in the following experiments and exclude BitFit as a submodule of UNIPELT. As for LoRA, there are a few setups where LoRA fails to learn effectively as well, such as STS-B and QQP ( K = { 100 , 500 } ), leading to high variance across runs. Apart from that, LoRA performs 5 Tuning other hyperparameters like learning rate does not appear to alleviate the issue either. quite competitively despite using fewer trainable parameters than methods like adapter, especially when K = 1000 , achieving the best or 2nd best performance on 4 of 8 tasks. As LoRA has a scaling factor  that can be seen as a static gating function under our formulation, we further investigate its importance by evaluating LoRA with different  . As shown in Fig. 3, LoRA is quite sensitive to the scaling factor and there seems to be no single optimal value that works well across multiple task and data setups. Such findings suggest that gating is critical and motivate us to use more fine-grained and dynamic control for UNIPELT. Besides, we observe that increasing  consistently results in faster convergence, possibly because the trainable parameters would receive larger gradient updates with a larger  . Next, we will turn to our proposed framework UNIPELT, which incorporates multiple existing PELT methods as submodules. Low-Resource Performance . Overall, UNIPELT (APL) and UNIPELT (AP) consistently achieve the best and second best average performance on both the development and test sets regardless of the number of training samples. The gains are generally 1~4% over the submodule that performs the best (when used individually). Such results demonstrate the advantages of our hybrid approach regarding model effectiveness and generalizability. At the per-task level, UNIPELT (APL) and UNIPELT (AP) perform the best or second best on 7/6/7 of 8 tasks when trained with 100/500/1,000 samples, and never perform the worst in any setup. When comparing the two variants, UNIPELT (APL) outperforms UNIPELT (AP) on 4/6/8 of 8 tasks when trained with 100/500/1,000 samples. Such results indicate that UNIPELT is quite robust and performs reliably under different scenarios. The improvements of UNIPELT over its submodules are generally larger when having fewer training samples, suggesting that UNIPELT performs especially well in the low-resource regime. In particular, on the tasks where other PELT methods fail to learn effectively such as CoLA and QQP ( K = 100 ), UNIPELT manages to achieve performance better than fine-tuning. UNIPELT vs. Upper Bound . In Table 2, we show the comparison of UNIPELT and the upper bound that takes the best performance of its submodules on each task. We observe that both UNIPELT (AP) and UNIPELT (APL) perform similarly or even better than their upper bound, which suggests that UNIPELT successfully learns to leverage different submodules and maintains (near) optimal performance under different setups. The fact that UNIPELT can outperform the upper bound also hints that a mixture of PELT methods (involving different parts of the PLM) might be inherently more effective than single methods (with a limited scope of the PLM architecture). High-Resource Performance . In Table 3, we list the performance of different methods when all training samples are used. UNIPELT again achieves the best overall performance. The gains are not as significant as in the low-resource setting, which is somewhat expected as existing PELT methods typically perform on par with fine-tuning given abundant training data and the potential of improvement is not as high. That said, the performance of UNIPELT is still the best or 2nd best on all 8 tasks, and generally comparable to the best submodule used individually on each task. Besides, simply combining multiple PELT methods without gating does not work well in the high-resource setting  although UNI PELT-NoGate never performs the worst in each task, its average performance is unsatisfactory (-0.89 vs. UNIPELT). We benchmark the efficiency of PELT methods and list in Table 4 their number of trainable parameters and training/inference time relative to fine-tuning. Parameter Efficiency . As the trainable parameters in PELT methods are almost negligible, combining multiple methods does not lead to significant losses in parameter efficiency. UNIPELT still has few trainable parameters compared to fine-tuning (0.99%~1.26%). The parameters can be further reduced if one uses more parameter-efficient variants ( e.g. , Karimi Mahabadi et al. (2021a)), which can be easily swapped with the vanilla version used in our current framework. Also, note that more trainable parameters do not always lead to better performance, as shown in our experiments and prior studies (He et al., 2021; Pfeiffer et al., 2021). Training and Inference Efficiency . Due to parameter efficiency, all PELT methods train 30%~50% faster than fine-tuning and incorporating multiple PELT methods into UNIPELT does not suffer from slower training. On the other hand, the inference time of PELT methods is generally longer since they involve more FLOPs. UNIPELT has a slightly larger inference overhead (4%~11% compared to its slowest submodule), which we argue is insignificant since larger models that may achieve similar performance gains ( e.g. , BERT large ) need around 300% inference time (Wolf et al., 2020). Parameter-Efficient Tuning of PLMs . As it is increasingly infeasible to train and store full copies of large PLMs for various downstream tasks, how to efficiently tune the PLMs with few trainable parameters becomes critical. Existing PELT methods can be largely divided into two categories based on whether new trainable parameters are introduced. Specifically, one may either train a subset of the model parameters such as the prediction head (Lee et al., 2019) and bias terms (Ben Zaken et al., 2021), or introduce task-specific parameters to different parts of the PLM such as before multi-head attention (Li and Liang, 2021) or after feedforward layer (Houlsby et al., 2019). As the number of PELT methods keeps increasing, the purpose of UNIPELT is to better understand and leverage the distinctions of various methods instead of proposing yet another method. Mixture-of-Experts . UNIPELT is also related to approaches that involve a high-capacity network and activate (upweight) different parts of the network given different inputs. One notable example is Mixture-of-Experts (MoE) (Jacobs et al., 1991; Shazeer et al., 2017), which maintains a set of experts (neural networks) and one or more trainable gates that select a combination of the experts specific to each input. Despite being conceptually similar, UNIPELT is different from MoE: the submodules in UNIPELT are not combined explicitly by summation like MoE but in sequential order and affect each other implicitly. Moreover, the experts are diverse in UNIPELT while usually homogeneous or identical in MoE methods. In this paper, we present a comprehensive study of representative parameter-efficient language model tuning (PELT) methods and propose a unified framework, which incorporates different PELT methods as submodules and learns to activate the most appropriate submodules for a given task or data setup. Our proposed framework consistently outperforms conventional fine-tuning as well as the submodules that it incorporates under different setups, and generally surpasses the upper bound that takes the best performance of each submodule used individually on each task. Our findings suggest that a mixture of multiple PELT methods that involve different parts of the PLM may be favorable regarding both model effectiveness and robustness. For future work, we will try to better understand the discrepancy of various PELT methods in different scenarios. We also plan to investigate a multi-task setting where multiple submodules can be activated and cooperate at the task level. We thank Xiang Lisa Li, Hai Ye, Rabeeh Karimi Mahabadi, Junxian He, Yiqing Xie, Yaqing Wang, and Liyuan Liu for helpful discussions and feedback. We thank anonymous reviewers for valuable comments and suggestions.
abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain method abstain abstain abstain abstain abstain result abstain method method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method abstain abstain abstain abstain method abstain abstain other abstain method abstain method abstain abstain method abstain abstain method method abstain method abstain abstain abstain abstain method abstain method abstain abstain method method method method result abstain method method abstain method method abstain method method method abstain method method abstain method method method method method abstain method abstain abstain method method abstain abstain abstain abstain method method method abstain method abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method result abstain abstain abstain result other other other other other other other other other other method abstain objective abstain abstain objective other other