Continual learning is essential for real-world deployment when there is a need to quickly adapt the model to new tasks without forgetting knowledge of old tasks. Existing work on continual sequence generation either always reuses existing parameters to learn new tasks, which is vulnerable to catastrophic forgetting on dissimilar tasks, or blindly adds new parameters for every new task, which could prevent knowledge sharing between similar tasks. To get the best of both worlds, in this work, we propose continual sequence generation with adaptive compositional modules to adaptively add modules in transformer architectures and compose both old and new modules for new tasks. We also incorporate pseudo experience replay to facilitate knowledge transfer in those shared modules. Experiment results on various sequences of generation tasks show that our framework can adaptively add modules or reuse modules based on task similarity, outperforming state-of-the-art baselines in terms of both performance and parameter efficiency. We make our code public at https://github.com/GT-SALT/ Adaptive-Compositional-Modules . Current state-of-the-art language generation models can achieve great performance on a wide range of sequence generation tasks (Radford et al., 2019; Lewis et al., 2020) with a static data distribution. However, real-world scenarios are often changing which requires the model to learn with dynamic data distributions. In such cases of data distributions shift, current generation models often suffer from catastrophic forgetting (Sun et al., 2019): models completely and abruptly forget previously learned information upon learning new information. Continual learning (CL) (Ring, 1998; Thrun, 1998) has been introduced to improve model's ability to learn tasks in a stream by mitigating forgetting Figure 1: Comparison between previous methods (a and and facilitating knowledge transfer (Lopez-Paz and Ranzato, 2017), however, continual sequence generation is relatively under-investigated. Comparing to continual learning on text classification and question answering (Wang et al., 2020; Holla et al., 2020; Huang et al., 2021), continual sequence generation is more challenging, since the output is no longer discrete labels but sequential text data in different styles/domains. Based on how to retain old knowledge while learning new tasks, current continual sequence generation methods can be categorized into two types. The first one continually learns new tasks on old parameters (Fig 1 a), with approaches like experience replay (Sun et al., 2019; Chuang et al., 2020) and regularization (Mi et al., 2020) to maintain old knowledge. However, since all tasks share the same parameters, some degree of interference between tasks is unavoidable. Another line of work continually inserts new task-specific modules (adapters proposed by Houlsby et al., 2019) into every transformer layer for every new task while freezing pretrained mod-3653 els and modules used by old tasks (Fig 1 b, Madotto et al., 2021), which might prevent knowledge transfer between tasks and introduce possible parameter redundancy. In this work, we aim to get the best of both worlds: how to encourage the models to reuse modules from previous tasks as much as possible and to only add new modules if needed? To this end, we propose continual sequence generation with adaptive compositional modules , as shown in Fig 1 c. Specifically, we introduce a two-stage process for every new coming task: a decision stage and a training stage. During decision stage, we decide which modules to reuse and whether we need to add a new module. During training stage, the model architecture is determined and fixed. We augment new task's training process with pseudo experience replay (Sun et al., 2019) to further mitigate forgetting and facilitate knowledge transfer in those shared layers. Our model architecture is adaptive , as it can automatically add new modules for dissimilar tasks and reuse modules for similar tasks, thus making it robust to different scenarios of continual learning. Furthermore, it is compositional because for every new task, our new architecture is composed of reused modules from old tasks and newly added modules, which allows knowledge reuse and transfer. To evaluate the above adaptive compositional framework, we experiment with four representative sequence generation tasks following prior work (Sun et al., 2019; Chuang et al., 2020): natural language generation, SQL query generation, summarization and task-oriented dialogue arriving in a stream. Different from prior work that only tests their methods on very short task sequences or long task sequences with similar tasks only, we validate our approach on longer sequences containing diverse tasks with different levels of similarity. We believe this is a suitable scenario to validate both the model's ability to mitigate forgetting and its ability to facilitate knowledge transfer. In summary, this work makes two key contributions: (1) We propose continual sequence generation with adaptive compositional modules, to maximize knowledge transfer via module-reusing while adaptively adding new modules to mitigate task-interference and catastrophic forgetting. (2) Experiments with longer and more task sequences show that our approach outperformed baselines with higher parameter efficiency. Continual Learning Without allocating new parameters for new tasks, prior work mainly leverages experience replay (Wang et al., 2019; Sun et al., 2019) and regularization to mitigate catastrophic forgetting. In experience replay, models are retrained on old examples from previous tasks while learning new tasks. Those old examples are usually stored in a fixed size (Mi et al., 2020) or expanding (Huang et al., 2021) memory buffer. Besides replaying old examples, regularization on the hidden states (Wang et al., 2019; Han et al., 2020; Huang et al., 2021) or parameters (Mi et al., 2020) could be further added to prevent severe distortion. Another line of work is to create new parameters for new tasks while freezing parameters used by old tasks. In computer vision, progressive neural network (Rusu et al., 2016) continually adds new branches of parameters for new image classification tasks with lateral connections to facilitate forward knowledge transfer. Dynamically expandable network (Yoon et al., 2017) expands neural networks at neuron level by using regularization to restrict the number of added neurons. While allocating a big network in advance, PackNet (Mallya and Lazebnik, 2018) continually assigns a parameter subset to each task by network pruning.Li et al. (2019) employ neural architecture search (Liu et al., 2018) to optimize on new task's structure before learning new tasks. In language domain, prior work often utilizes adapter (Houlsby et al., 2019; Madotto et al., 2021; Ermis et al., 2022), which could be considered as task-specific MLPs inserted into frozen transformer layers. However, since all adapter modules are designed for only one specific task, no knowledge transfer is directly allowed in this case. Extra modules like attention module (Pfeiffer et al., 2021), capsule network (Ke et al., 2021), and hypernetworks (Jin et al., 2021) are demonstrated beneficial for knowledge transfer, but they need to introduce extra parameters and fail to consider any reusable or compositional modules. Avoiding privacy concerns, this work also follows a line of work that doesn't store real examples for experience replay, such as generating examples by GAN (Atkinson et al., 2018), synthesizing examples (Xu et al., 2022) by model-inversion (Smith et al., 2021b), and using unlabeled data in the learning environment (Smith et al., 2021a). In language domain, LAMOL (Sun et al., 2019) trains the language model to solve current tasks and generate 3654 current training examples simultaneously, then this model can generate pseudo old examples for replay before any new tasks. We adopt this pseudo experience replay along to alleviate the forgetting in the shared modules of our approach. Continual Learning for Sequence Generation Building on an auto-regressive language model, LAMOL (Sun et al., 2019) makes initial exploration on continual sequence generation. On the basis of LAMOL, knowledge distillation (Chuang et al., 2020; Sun et al., 2020) is shown to be effective via improving knowledge transfer while changing tasks. ARPER (Mi et al., 2020) combines regularization on parameters (Kirkpatrick et al., 2017) with prioritized exemplar replay. Keeping the pretrained model frozen, Madotto et al. (2021) added task-specific modules for each task together with a perplexity-based classifier, without taking into account the potential for knowledge transfer between different tasks. Instead of blindly adding new modules for new tasks, our approach can detect reusable modules and strategically add new adapter modules in those layers in which reusing old modules would lead to severe forgetting. Without introducing extra knowledge transfer modules, our approach enables knowledge transfer via module sharing. Task-specific Modules Traditional finetuning approaches (Peters et al., 2018; Devlin et al., 2019; Radford et al., 2019) usually modify all the parameters in large pretrained modules while learning downstream tasks. Recently, a line of work has been proposed to improve the parameter-efficiency of finetuning by inserting task-specific modules into freezing pretrained models. Adapter (Houlsby et al., 2019) inserts MLP layers into each transformer layer. PrefixTuning (Li and Liang, 2021) prepends key-value pairs to each transformer layer as activations. Prior work also shows that these task-specific modules might benefit from a more adaptive usage. For example, AdapterDrop (Rckl et al., 2021) shows that removing adapters from lower transformer layers can almost maintain the original performance while reducing computational overhead. Guo et al. (2021) leveraged latent variables to decide whether to skip adapter modules in certain transformer layers to speed up decoding. However, our approach goes beyond the notion of task-specific, recomposes reusable modules from different tasks, and learns compositional architectures for new coming tasks. Continual Generation Formulation Assuming multiple sequence generation tasks { T 1 ...T n } arrive in a stream, each task T i has a set of training examples { P i 1 , P i 2 ..., P ik } , where P ij denotes a ( input, output ) pair in Task i . While learning on task T i ( i > 2) , we have no access to examples from previous tasks. The final goal is to optimize the model's average performance on all tasks after training on the whole sequence. Finetuning In order to integrate different sequence generation tasks into a single framework, we use finetuning as a general strategy. On the basis of an autoregressive language model, the core idea is to feed the model input and train the model to generate the corresponding output subsequently. To distinguish between tasks, we add an extra question following every input to describe the purpose of each task. For example, the question for natural language generation tasks is What is the natural language form? Formally, for each ( input, question, output ) triple, the model is optimized to generate the corresponding output given input and question : L finetune ( x ) = n (cid:88) t = m +1  log P ( x t | x <t ) where x = { x 1 , ..., x n } denotes the concatenation of input , question and output , and { x 1 , ..., x m } refers to input and question . Adapter The module used in our framework refers to adapter (Houlsby et al., 2019), which is a task-specific module inserted into each frozen pretrained transformer layers (Vaswani et al., 2017). In addition to residual connection (He et al., 2016) and layer normalization (Ba et al., 2016), one transformer layer contains two primary sub-layers: an attention layer and a feed forward layer. One adapter module consists of two multi-layer perceptrons ( MLP ), one ( MLPMH ) following the multi-head attention layer and one ( MLPFF ) following the feed forward layer. Motivated by prior continual sequence generation work (Madotto et al., 2021) that uses Adapter (Houlsby et al., 2019) to insert new adapter module into every transformer layer for each new coming task, we propose to strategically decide whether we can reuse some adapter modules from old tasks before training on each new coming task, in a two-stage manner: decision stage and training stage, where the former determines the architecture for new tasks and the later trains the model. The decision stage aims to answer two questions: do we need to add a new module in this layer? If not, which old modules should we reuse? Inspired by interpolation-based data augmentation (Chen et al., 2020, 2021) and neural architecture search (Liu et al., 2018), we utilize Hidden State Mixing for module selection. Assume that there are several modules as potential candidates to be selected, after calculating their output separately, we calculate their weighted average as the overall output, which is then passed to the next part of the model (See the left part in Figure 2). After training the entire model end-to-end, we assume that the module with the largest learned weight is the most useful one, and thus will be selected for the reuse. Formally, assume that we already have inserted k modules into the l th transformer layer, each consisting of two MLPs: ( MLP 1 ,lMH , MLP 1 ,lFF ) ... ( MLP k,lMH , MLP k,lFF ) . At the beginning of decision stage, we add one more module ( MLP k +1 ,l MH , MLP k +1 ,l FF ) . Given these learnable weight coefficients [  1 ,l , . . . ,  k +1 ,l ] , multi-head attention layer output o lmh , the feed forward layer output o lff , we mix the hidden states as follow: h lmh = k +1 (cid:88) t =1  t,l MLP t,lMH ( o lmh ) h l ff = k +1 (cid:88) t =1  t,l MLP t,l FF ( o l ff ) where both h lmh and h lff are then fed into their following Add & Norm layers. To ensure (cid:80) k +1 t =1  t,l = 1 , we use softmax function to produce  1 ,l , . . . ,  k +1 ,l from c 1 ,l , . . . , c k +1 ,l :  i,l = e c i,l (cid:80) k +1 t =1 e c t,l , i = 1 . . . k + 1 Using this mixing approach in every transformer layer, we optimize our model using L train (see Sec 4.2) for the new task and find the most suitable modules for each layer. Note that (i) In this process, the pretrained model and all old modules are frozen, and only mixing coefficients and newly added modules will be learned. (ii) Calculating the weighted average is a convenient approximation of using one adapter at a time, which is the real setting during training stage and inference. (iii) Comparing to other baselines in Figure 1, introduced decision stage to decide the architecture does introduce extra computation, while computation of different MLPs at one position is parallelizable to speed up. To avoid the learned weight coefficient  1 ,l , . . . ,  k +1 ,l to be too close to a uniform distribution in certain layers, we further add an additional regularization term to L train , which is the sum of entropy of every discrete probability distribution [  1 ,l , . . . ,  k +1 ,l ] : L entropy =  (cid:88) l k +1 (cid:88) i =1   i,l log(  i,l ) where  is a coefficient tuned as a hyper-parameter. In this stage, a trivial solution could be allocating a new module in every layer regardless of whether old modules are reusable. To avoid this trivial solution and reuse shareable modules as much as possible, we design a prior using the initialization of the coefficient weights . For every l , c 1 ,l ...c k,l is initialized to c ( c > 0) , while c k +1 ,l is initialized to  c . After softmax , the weight of each old module is e 2 c times the weight of the new module, increasing the tendency to reuse old modules. We further incorporate pseudo experience replay (Sun et al., 2019) to mitigate forgetting and facilitate knowledge transfer in those shared modules. The main idea is to teach a generative model to solve current task and to generate current task's examples simultaneously . Then before training on each new task, we can generate a set of pseudo old examples and replay them during training. Thus, in addition to the finetuning loss to solve each task, we introduce an extra loss L gen for the model to generate current task's examples. Formally, given the whole sequence of x = { input, question, output } , we first add a special token [GEN] at the beginning of x to form a new sequence x (cid:48) , and then optimize the model as follows: L gen ( x (cid:48) ) = n +1 (cid:88) t =1  log P ( x (cid:48) t | x (cid:48) <t ) 3656 Figure 2: Our proposed model architecture with adaptive compositional modules for transformer layers. Note that we use different special tokens for different tasks, thus we can generate examples for specified tasks afterwards. Combining with the finetune loss, the overall training loss is: L train = L finetune + L gen where  is the weight for the L gen loss. Once our model has the ability to generate pseudo examples from old tasks, another question is When to generate pseudo examples? Since those pseudo examples are for shared modules between old tasks and the current task, we only generate them while some old modules are reused for the current task. In that case, we train our model using L train on the current dataset together with the generated examples. Otherwise, there is no need for pseudo experience replay and we just train our model using L train on the current dataset. 5 Experiments 5.1 Datasets Following Sun et al. (2019) and Chuang et al. (2020), we evaluate our approach on four representative sequence generation tasks: natural language generation, SQL query generation, summarization and task-oriented dialogue modeling. Specifically, we test our proposed approach under two common scenarios: (1) CL on similar tasks : in this case, the new coming tasks often share the same task pattern with learned tasks, but are from different domains. We use E2ENLG (Novikova et al., 2017) and four different domains (restaurant, hotel, tv, laptop) from RNNLG (Wen et al., 2015) to form five similar tasks. Then we use four different orders of these tasks as our testing task sequences. (2) CL on dissimilar tasks : in this case, the distribution shift between new tasks and old tasks could be relatively large, so the major challenge is to retain old knowledge as much as possible while learning new tasks. In this case, we further incorporate WikiSQL (SQL query generation, Zhong et al., 2017), CNN/DailyMail (news article summarization See et al., 2017), MultiWOZ (semantic state sequence generation (Budzianowski et al., 2018)) into our task sequences 1 . We randomly pick four different orders as our testing task sequences. In total, we use eight different task sequences (Table 1) to evaluate our models. The statistics/metrics for each dataset and the finetuing results are in Appendix A. 1 We use e2e for E2ENLG, rest for RNNLG (restau-rant), hotel for RNNLG (hotel), tv for RNNLG (tv), laptop for RNNLG (laptop), wiki for WikiSQL, cnn for CNN/DailyMail, woz for MultiWOZ. We compare our proposed model with the following baselines: (i) Finetune (Yogatama et al., 2019): We finetuned GPT-2 model on several tasks sequentially. (ii) EWC (Kirkpatrick et al., 2017) added regularization on parameters according to their importance to old tasks. (iii) LAMOL (Sun et al., 2019) finetuned the whole GPT-2 model continually with the help of pseudo experience replay. (iv) Adapter+CL (Madotto et al., 2021) inserted adapter (Houlsby et al., 2019) modules into every GPT-2's layer for each task. (v) Adapter+Drop (Rckl et al., 2021): We removed all those adapter modules from the first three layers in GPT-2 based on Adapter+CL. (vi) Adapter+LAMOL : We only inserted adapter modules into every transformer layer for the first task, then used those adapter modules to learn the whole the task sequence with pseudo experience replay. Note that ARPER (Mi et al., 2020) also tackles the problem of continual sequence generation, but it needs an extra memory buffer to store examples from old tasks, which is not comparable with ours. Implementation Details We use GPT-2 (Rad-ford et al., 2019) in HugginceFace Transformers (Wolf et al., 2020) as our backbone and adapter implementation by AdapterHub (Pfeiffer et al., 2020). More details can be found in Appendix A. 6 Results and Analysis To evaluate the overall performance on all tasks, we use the mean of all tasks' performance score following Sun et al. (2019); Mi et al. (2020); Madotto et al. (2021). For each scenario ( similar tasks and dissimilar tasks), we report the average of mean scores on all sequences as an overall metric. Beyond these, we also provide (i) evaluation results using geometric mean and (ii) final performance of each task in Appendix A. Table 2 summarizes the final performance on all eight task sequences. We observed that finetuning sequentially suffered from very severe forgetting, no matter on similar or dissimilar tasks, highlighting the importance of continual learning work. Though EWC can significantly increase the performance of finetuning, its performance is still far behind LAMOL, highlighting the importance of experience replay. For sequences containing similar tasks, the performance of Adapter+CL is inferior to Adapter+LAMOL even with more learnable parameters. This indicates that sharing parameters and experience replay can further facilitate knowledge transfer when tasks are similar. On the premise of pseudo experience replay, our method performs better than Adapter+LAMOL, demonstrating the effectiveness of our adaptive compositional architecture. Our approach also achieves a much higher parameter efficiency than Adapter+CL and Adapter+Drop. For sequences containing dissimilar tasks where the transferable knowledge is limited and parameter sharing might cause degradation, Adapter+CL and Adapter+Drop seem more robust compared to Adapter+LAMOL and LAMOL, since they avoid catastrophic forgetting by parameter isolation. Using a similar number of parameters to Adapter+Drop, our method outperforms Adapter+CL consistently on all task sequences, confirming that our method can prevent interference between dissimilar tasks while reducing parameter redundancy. We randomly selected task sequence #1 from similar tasks and sequence #8 from sequences of dissimilar tasks for our ablation studies. Importance of Each Component To examine the importance of each component in our method, we experiment with different settings: not using entropy loss (w/o Entropy Loss), initializing all weight coefficients with zero (w/o Weight Ini), and not replaying pseudo data (w/o Pseudo ER). As shown in Table 3, we found that (i) After removing entropy loss, the performance on sequence #1 is almost maintained by using more parameters. Meanwhile, the performance on sequence #8 drops significantly while using the same number of parameters. This observation suggests that the en-3658 Methods Finetune EWC LAMOL Adapter+CL Adapter+Drop Adapter+LAMOL Ours PseudoExperience Replay (cid:55) (cid:55) (cid:51) (cid:55) (cid:55) (cid:51) (cid:51) Similar Tasks # 1 43.0 56.9 66.3 64.2 63.9 65.9 66.1 # 2 37.0 47.9 67.0 64.2 63.9 66.2 66.5 # 3 51.7 61.4 66.6 64.2 63.9 65.6 65.8 # 4 45.0 58.3 66.6 64.2 63.9 65.2 65.7 Avg Performance 44.2 56.2 66.6 64.2 63.9 65.7 66.0 Avg Learnable Para. tropy loss is beneficial to achieve a better trade-off between adding parameters and maintaining good performance. (ii) When we initialize all weight coefficients with zero, there is no explicit tendency to reuse old examples. In this case, many redundant modules are created thus preventing knowledge transfer, which leads to performance drop on both sequences. The drop on sequence #1 is more severe due to there is more transferable knowledge between similar tasks. We therefore conclude that weight initialization is important to enable knowledge transfer between similar tasks. (iii) Removing pseudo experience replay leads to the most severe performance drop on both sequences. Though our approach strategically detect which modules can be reused, directly training them on new tasks without protecting old knowledge will lead to catastrophic Length Adapter+CL Adapter+LAMOL Ours 2 Tasks(#1) 56.8 (+0.0) 57.5 (+0.8) 57.7 (+0.9) 3 Tasks(#1) 59.5 (+0.0) 60.3 (+0.6) 60.1 (+0.5) 4 Tasks(#1) 62.3 (+0.0) 63.5 (+1.3) 63.7 (+1.6) 5 Tasks(#1) 64.2 (+0.0) 65.9 (+2.0) 66.1 (+2.1) 2 Tasks(#8) 45.4 (+0.0) 46.2 (+1.3) 46.0 (+1.2) 3 Tasks(#8) 51.3 (+0.0) 51.9 (+0.8) 52.3 (+0.9) 4 Tasks(#8) 50.9 (+0.0) 49.7 (-1.7) 51.8 (+0.6) 5 Tasks(#8) 57.3 (+0.0) 53.8 (-4.6) 58.2 (+0.5) Table 4: Impact of the task sequence length. Impact of Task Sequence Length Prior work in continual learning (Madotto et al., 2021; Huang et al., 2021) suggests that the differences in sequence length could influence the performance of continual learning. To this end, we further investigated the impact of sequence length in Table 4, where we reported the average performance at every step and calculated Backward Transfer following Lopez-Paz and Ranzato (2017): BW T k = 1 k  1 E i =1 ...k  1 ( R k,i  R i,i ) where R i,j is the performance score on the j th task after training on the i th task. We found that, on sequence #1, Adapter+LAMOL and our method consistently outperform Adapter+CL in all stages, which could be explained by better knowledge transfer between multiple tasks. Beyond that, our method outperforms Adapter+LAMOL in most cases, demonstrating the benefits of adaptively adding modules. On sequence #8, Adapter+LAMOL struggles when the length of task sequence becomes longer. As more and more tasks arrive, the impact of task dissimilarity and distribution shift gets larger that pseudo experience replay cannot cope with. In that case, there is limited backward transfer but severe forgetting. In contrast, Adapter+CL and our method demonstrate their robustness after learning more tasks in a stream. Our method also outperforms Adapter throughout the learning process, demonstrating we can enable knowledge transfer even the similarity between tasks is limited. Case Study We selected e2e in sequence #1 and wiki in sequence #8 as two representative tasks to illustrate the final output generated by different approaches in Table 5. After training on the whole sequence, Adapter+LAMOL cannot correctly convey the information provided in the input, suffering from generating grammar mistakes and missing key points. This could be attributed to the interference from learning new coming tasks. While Adapter+CL successfully mitigate this problem by parameter isolation, our approach works similarly using less parameters and generates better sequences without redundant information. To illustrate the process of adding/reusing modules, we depict the model architecture at each stage in Fig 3 using sequence #4, which is the most challenging sequence containing similar tasks according to Table 2. Since the similarity between the second task (e2e) and the first task (hotel) is low (see Figure 4 in Appendix A), our framework automatically learns to add extra adapter modules in layer { 6 , 8 , 9 , 10 , 11 } before training on the second task. When the third task (rest) arrives, given its high similarity to the first task, our method correctly decides to reuse all modules used in the first task. Interestingly, the architecture for the fourth task is composed of shared modules with the first 3 tasks in layer { 1 , 2 , 3 , 4 , 5 , 7 , 12 } , shared module with the second task in layer 6 , shared the mod-3660 E2E NLG (#1): name[Strada], eatType[coffee shop], area[city centre] Reference There is a coffee shop in the city centre called the Strada. ule with the first and the third task in layer 8 , and also added new modules for the fourth task in layer { 9 , 10 , 11 } . For the fifth task, our method reuses all modules used by the fourth tasks due to their high similarity. This demonstrates that our method is adaptive to different incoming tasks and is able to compose modules from different old tasks for new tasks. We also provide a comparison in Appendix B to demonstrate the effect of reusing modules from different transformer layers. This work examined continual sequence generation with adaptive compositional modules, where we proposed hidden state mixing to adaptively compose old and new modules for new tasks and utilized pseudo experience replay to facilitate knowledge transfer. Experiments conducted on various sequence generation tasks demonstrated that our method achieves better performances with higher parameter efficiency over previous state-of-the-art baselines, both on similar task sequences and dissimilar task sequences. Our work is also subject to a few limitations such as the introduced extra training time. In the future, we plan to investigate how to further speed up the decision stage more efficiently and generalize the current framework to more diverse NLP tasks such as text classification and machine translation. We would like to thank the anonymous reviewers for their helpful comments, and the members of Georgia Tech SALT group for their feedback. This work is funded in part by Salesforce and Cisco.
abstain abstain objective abstain result other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective objective objective abstain abstain objective objective method abstain method objective result other other other other other other other other other other other method other method other other other other objective abstain other other other other other other other objective other abstain other method other objective other other method other other abstain method method method method method method method method method method abstain abstain abstain abstain method abstain method abstain abstain method abstain method method method method abstain method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain other abstain method abstain abstain abstain abstain other abstain other abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain method abstain abstain objective objective abstain objective other other