Recent studies have shown remarkable success in end-to-end task-oriented dialog system. However, most neural models rely on large training data, which are only available for a certain number of task domains, such as navigation and scheduling. This makes it difficult to scalable for a new domain with limited labeled data. However, there has been relatively little research on how to effectively use data from all domains to improve the performance of each domain and also unseen domains. To this end, we investigate methods that can make explicit use of domain knowledge and introduce a shared-private network to learn shared and specific knowledge. In addition, we propose a novel Dynamic Fusion Network (DF-Net) which automatically exploit the relevance between the target domain and each domain. Results show that our model outperforms existing methods on multi-domain dialogue, giving the state-of-the-art in the literature. Besides, with little training data, we show its transferability by outperforming prior best model by 13.9% on average. Task-oriented dialogue systems (Young et al., 2013) help users to achieve specific goals such as restaurant reservation or navigation inquiry. In recent years, end-to-end methods in the literature usually take the sequence-to-sequence (Seq2Seq) model to generate a response from a dialogue history (Eric and Manning, 2017; Eric et al., 2017; Madotto et al., 2018; Wen et al., 2018; Gangi Reddy et al., 2019; Qin et al., 2019b; Wu et al., 2019a). Taking the dialogue in Figure 1 as an example, to answer the driver's query about the  gas station , the end-to-end dialogue system directly generates system response given the query and a corresponding knowledge base (KB). Though achieving promising performance, end-to-end models rely on a considerable amount of labeled data, which limits their usefulness for new and extended domains. In practice, we cannot col-lect rich datasets for each new domain. Hence, it is important to consider methods that can effectively transfer knowledge from a source domain with suf-ficient labeled data to a target domain with limited or little labeled data. Existing work can be classified into two main categories. As shown in Figure 2(a), the first strand of work (Eric and Manning, 2017; Eric et al., 2017; Madotto et al., 2018; Wu et al., 2019a) simply combines multi-domain datasets for training. Such methods can implicitly extract the shared features but fail to effectively capture domain-specific knowledge. As shown in Figure 2(b), The second strand of work (Wen et al., 2018; Qin et al., 2019b) trains model separately for each domain, which can better capture domain-specific features. However, those methods ignore shared knowledge between different domains (e.g. the location word exists in both schedule domain and navigation domain). We consider addressing the limitation of existing work by modeling knowledge connections between domains explicitly. In particular, a simple baseline NoDomain-SpecificFeatures AModule BModule CModule SharedModule NoDomain-SharedFeatures SharedModule AModule BModule CModule DynamicDomain-SpecificFusion Shared-SpecificFusion to incorporate domain-shared and domain-private features is shared-private framework (Liu et al., 2017; Zhong et al., 2018; Wu et al., 2019b). Shown in Figure 2(c), it includes a shared module to capture domain-shared feature and a private module for each domain. The method explicitly differentiates shared and private knowledge. However, this framework still has two issues: (1) given a new domain with extremely little data, the private module can fail to effectively extract the corresponding domain knowledge. (2) the framework neglects the fine-grained relevance across certain subsets of domains. (e.g. schedule domain is more relevant to the navigation than to the weather domain.) To address the above issues, we further propose a novel Dynamic Fusion Network (DF-Net), which is shown in Figure 2 (d). In contrast to the shared-private model, a dynamic fusion module (see  2.3) is further introduced to explicitly capture the correlation between domains. In particular, a gate is leveraged to automatically find the correlation between a current input and all domain-specific models, so that a weight can be assigned to each domain for extracting knowledge. Such a mechanism is adopted for both the encoder and the decoder, and also a memory module to query knowledge base features. Given a new domain with little or no training data, our model can still make the best use of existing domains, which cannot be achieved by the baseline model. We conduct experiments on two public benchmarks, namely SMD (Eric et al., 2017) and MultiWOZ 2.1 (Budzianowski et al., 2018). Results show that our framework consistently and significantly outperforms the current state-of-the-art methods. With limited training data, our framework outperforms the prior best methods by 13.9% on average. To our best of knowledge, this is the first work to effectively explore shared-private framework in multi-domain end-to-end task-oriented dialog. In addition, when given a new domain which with few or zero shot data, our extended dynamic fusion framework can utilize fine-grained knowledge to obtain desirable accuracies, which makes it more adaptable to new domains. All datasets and code are publicly available at: https://github.com/LooperXX/DF-Net . We build our model based on a seq2seq dialogue generation model (  2.1), as shown in Figure 3(a). To explicitly integrate domain awareness, as shown in Figure 3(b) we first propose to use a shared-private framework (  2.2) to learn shared and the corresponding domain-specific features. Next, we further use a dynamic fusion network (  2.3) to dynamically exploit the correlation between all domains for fine-grained knowledge transfer, which is shown in Figure 3(c). In addition, adversarial training is applied to encourage shared module generate domain-shared feature. We define the Seq2Seq task-oriented dialogue generation as finding the system response Y according to the input dialogue history X and KB B . Formally, the probability of a response is defined as p ( Y | X, B ) = n (cid:89) t =1 p ( y t | y 1 , ..., y t  1 , X, B ) , (1) where y t represents an output token. In a vanilla Seq2Seq task-oriented dialogue system (Eric and Manning, 2017), a long short-term Memory network (LSTM, Hochreiter and Schmidhuber (1997)) is used to encode the dialogue history X = ( x 1 , x 2 , .., x T ) ( T is the number of tokens in the dialogue history) to produce shared context-sensitive External Knowledge(EK) START EKDistribution Domain+START +  "#$,&' VocabularyDistribution SharedEncoder PrivateEncoder Shared-SpecificFusion External Knowledge(EK) EKDistribution + VocabularyDistribution SharedEncoder Fusion DynamicFusion External Knowledge(EK) + Fusion PrivateEncoder SharedEncoder Domain+START EKDistribution VocabularyDistribution hidden states H = ( h 1 , h 2 , ..., h T ) : h i = BiLSTM enc (cid:16)  emb ( x i ) , h i  1 (cid:17) , (2) where  emb (  ) represents the word embedding matrix. LSTM is also used to repeatedly predict outputs ( y 1 , y 2 , ..., y t  1 ) by the decoder hidden states ( h dec , 1 , h dec , 2 , ..., h dec ,t ) . For the generation of y t , the model first calculates an attentive representation h (cid:48) dec ,t of the dialogue history over the encoding representation H . Then, the concatenation of h dec ,t and h (cid:48) dec ,t is projected to the vocabulary space V by U : o t = U [ h dec ,t , h (cid:48) dec ,t ] , (3) where o t is the score (logit) for the next token generation. p ( y t | y 1 , ..., y t  1 , X, B ) = Softmax ( o t ) . (4) Different from typical text generation with Seq2seq model, the successful conversations for task-oriented dialogue system heavily depend on accurate knowledge base (KB) queries. We adopt the global-to-local memory pointer mechanism (GLMP) (Wu et al., 2019a) to query the entities in KB, which has shown the best performance. An external knowledge memory is proposed to store knowledge base (KB) B and dialogue history X . The KB memory is designed for the knowledge source while the dialogue memory is used for directly copying history words. The entities in external knowledge memory are represented in a triple format and stored in the memory module, which can be denoted as M = [ B ; X ] = ( m 1 , . . . , m b + T ) , where m i is one of the triplet of M , b and T denotes the number of KB and dialog history respectively. For a k -hop memory network, the external knowledge is composed of a set of trainable embedding matrices C = ( C 1 , . . . , C k +1 ) . We can query knowledge both in encoder and decoder process to enhance model interaction with knowledge module. In addition, it can loop over k hops and compute the attention weights at each hop k using where c ki is the embedding in i th memory position using the embedding matrix C k . We obtain the global memory pointer G = ( g 1 , . . . , g b + T ) by applying g ki = Sigmoid(( q k enc ) (cid:62) c ki ) , which is used to filter the external knowledge for relevant information for decoding. Finally, the model reads out the memory o k by the weighted sum over c k +1 and updates the query vector q k +1 enc . Formally, o k enc = (cid:88) i p ki c k +1 i , q k +1 enc = q k enc + o k enc . q k +1 enc can be seen as the encoded KB information, and is used to initialized the decoder. Query Knowledge in Decoder we use a sketch tag to denote all the possible slot types that start with a special token. (e.g., @address stands for all the Address ). When a sketch tag is generated by Eq. 4 at t timestep, we use the concatenation of the hidden states h dec ,t and the attentive representation h (cid:48) dec ,t to query knowledge, which is similar with SharedModule AModule BModule CModule Expert Gate MLP Feature Extractor DynamicDomain-SpecificFusion Shared-PrivateFusion SharedModule AModule BModule CModule Expert Gate Shared-PrivateFusion Feature Extractor DynamicDomain-SpecificFusion Shared-PrivateFusion SharedModule AModule BModule CModule Expert Gate MLP Feature Extractor DynamicDomain-SpecificFusion Shared-SpecificFusion ( &,) ( &,* ( &,+ Figure 4: The dynamic fusion layer for fusing domain-shared feature and domain-specific feature. the process of querying knowledge in the encoder: q 1 dec = [ h dec ,t , h (cid:48) dec ,t ] , (8) p ki = Softmax (( q k dec ) (cid:62) c ki g ki ) . Here, we can treat P t = ( p k 1 ,. . . , p kb + T ) as the probabilities of queried knowledge, and select the word with the highest probability from the query result as the generated word. The model in section 2.1 is trained over mixed multi-domain datasets and the model parameters are shared across all domains. We call such model as shared encoder-decoder model. Here, we propose to use a shared-private framework including a shared encoder-decoder for capturing domain-shared feature and a private model for each domain to consider the domain-specific features explicitly. Each instance X goes through both the shared and its corresponding private encoder-decoder. Enhancing Encoder Given an instance along with its domain, the shared-private encoder-decoder generates a sequence of encoder vectors denoted as H { s,d } enc , including s hared and d omain-specific representation from corresponding encoder: H { s,d } enc =( h { s,d } enc , 1 , . . . , h { s,d } enc ,T ) =BiLSTM { s,d } enc ( X ) . In addition, self-attention has been shown useful for obtaining context information (Zhong et al., 2018). Finally, we follow Zhong et al. (2018) to use self-attention over H f enc to get context vector c f enc . We replace h T with c f enc in Eq. 5. This makes our query vector combine the domain-shared feature with domain-specific feature. We also apply the shared-specific fusion function to the hidden states and the mixture vector is: shprivate : ( h s dec ,t , h d dec ,t )  h f dec ,t . Similarly, we obtain the fused attentive representation h f (cid:48) dec,t by applying attention from h f dec ,t over H f enc . Finally, we replace [ h dec ,t , h (cid:48) dec ,t ] in Eq. 8 with [ h f dec ,t , h f (cid:48) dec ,t ] which incorporates shared and domain-specific features. The shared-private framework can capture the corresponding specific feature, but neglects the fine-grained relevance across certain subsets of domains. We further propose a dynamic fusion layer to explicitly leverage all domain knowledge, which is shown in Figure 4. Given an instance from any domain, we first put it to multiple private encoder-decoder to obtain domain-specific features from all domains. Next, all domain-specific features are fused by a dynamic domain-specific feature fusion module, followed by a shared-specific feature fusion for obtaining shared-specific features. Dynamic Domain-Specific Feature Fusion Given domain-specific features from all domains, a Mixture-of-Experts mechanism (MoE) (Guo et al., 2018) is adapted to dynamically incorporate all domain-specific knowledge for the current input in both encoder and decoder. Now, we give a detailed description on how to fuse the timestep t of decoding and the fusion process is the same to encoder. Given all domain feature representations in t decoding steps: { h d i dec ,t } | D | i =1 , where | D | represents the number of domains, an expert gate E takes { h d i dec ,t } as input and outputs a softmax score  t,i that represents the degree of correlation between each domain and the current input token. We achieve this by a simple feedforward layer:  t = Softmax ( W  h d dec ,t + b ) . The final domain-specific feature vector is a mixture of all domain outputs, dictated by the expert gate weights  t = (  t, 1 , . . . ,  t, | D | ) , which can be written as h d f dec ,t = (cid:80) i  t,i h d i dec ,t . During training, take the decoder for example, we apply the cross-entropy loss L moe dec as the supervision signal for the expert gate to predict the domain of each token in the response, where the expert gate output  t can be treated as the t th to-ken's predicted domain probability distribution by multiple private decoder. Hence, the more accurate the domain prediction is, the more correct expert gets: L moe dec =  n (cid:88) t =1 | D | (cid:88) i =1 ( e i  log(  t,i |  s ,  mdec )) , (16) where  s represents the parameters of encoder-decoder model,  mdec represents the parameters of the MoE module (Eq. 15) in the decoder and e i  { 0 , 1 } represents whether the response with n tokens belongs to the domain d i . Similarly, we can get the L moe enc for the encoder and sum up them as: L moe = L moe enc + L moe dec . L moe is used to encourage samples from a certain source domain to use the correct expert, and each expert learns corresponding domain-specific features. When a new domain has little or no labeled data, the expert gate can automatically calculate the correlation between different domains with the target domain and thus better transfer knowledge from different source domains in both encoder and decoder module. Shared-Specific Feature Fusion We directly apply shprivate operation to fuse shared and final domain-specific feature: shprivate : ( h s dec ,t , h d f dec ,t )  h f dec ,t . Finally, we denote the dynamic fusion function as dynamic( h s dec,t , { h d i dec,t } | D | i =1 ) . Similar to Section 2.2, we replace [ h dec ,t , h (cid:48) dec ,t ] in Eq. 8 with [ h f dec ,t , h f (cid:48) dec ,t ] . The other components are kept the same as the shared-private encoder-decoder framework. Adversarial Training To encourage the model to learn domain-shared features, we apply adversarial learning on the shared encoder and decoder module. Following Liu et al. (2017), a gradient reversal layer (Ganin and Lempitsky, 2014) is introduced after the domain classifier layer. The adversarial training loss is denoted as L adv . We follow Qin et al. (2019a) and the final loss function of our Dynamic fusion network is defined as: L =  b L basic +  m L moe +  a L adv , (18) where L basic keep the same as GLMP (Wu et al., 2019a),  b ,  m and  a are hyper-parameters. More details about L basic and L adv can be found in appendix. Two publicly available datasets are used in this paper, which include SMD (Eric et al., 2017) and an extension of Multi-WOZ 2.1 (Budzianowski et al., 2018) that we equip the corresponding KB to every dialogue. 1 The detailed statistics are also presented in Table 1. We follow the same partition as Eric et al. (2017), Madotto et al. (2018) and Wu et al. (2019a) on SMD and (Budzianowski et al., 2018) on Multi-WOZ 2.1. The dimensionality of the embedding and LSTM hidden units is 128 . The dropout ratio we use in our framework is selected from { 0 . 1 , 0 . 2 } and the batch size from { 16 , 32 } . In the framework, we adopt the weight typing trick (Wu et al., 2019a). We use Adam (Kingma and Ba, 2015) to optimize the parameters in our model and adopt the suggested hyper-parameters for optimization. All hyper-parameters are selected according to validation set. More details about hyper-parameters can be found in Appendix. We compare our model with the following state-of-the-art baselines. Mem2Seq (Madotto et al., 2018): the model takes dialogue history and KB entities as input and uses a pointer gate to control either generating a vocabulary word or selecting an input as the output. DSR (Wen et al., 2018): the model leverages dialogue state representation to retrieve the KB implicitly and applies copying mechanism to retrieve entities from knowledge base while decoding. KB-retriever (Qin et al., 2019b): the model adopts a retriever module to retrieve the most relevant KB row and filter the irrelevant information for the generation process. GLMP (Wu et al., 2019a): the framework adopts the global-to-local pointer mechanism to query the knowledge base during decoding and achieve state-of-the-art performance. For Mem2Seq , DSR , KB-retriever 2 , we adopt the reported results from Qin et al. (2019b) and Wu et al. (2019a). For GLMP , we rerun their public code to obtain results on same datasets. 3 2 For Multi-WOZ 2.1 dataset, most dialogs are supported by more than single row, which can not processed by KBretriever , so we compare our framework with it in SMD and Camrest datasets. 3 Note that, we find that Wu et al. (2019a) report macro entity F1 as the micro F1, so we rerun their models (https://github.com/jasonwu0731/GLMP) and obtain results. Follow the prior work (Eric et al., 2017; Madotto et al., 2018; Wen et al., 2018; Wu et al., 2019a; Qin et al., 2019b), we adopt the BLEU and Micro Entity F1 metrics to evaluate model performance. The results on the two datasets are shown in Table 2, we can observe that: 1) The basic shared-private framework outperforms the best prior model GLMP in all the datasets. This indicates that the combination of domain-shared and domain-specific features can better enhance each domain performanc compared with only utilizing the implicit domain-shared features. 2) Our framework achieves the state-of-the-art performance on two multi-domain task-oriented dialog datasets, namely SMD and Multi-WOZ 2.1. On SMD dataset, our model has the highest BLEU compared with baselines, which shows that our framework can generate more fluent response. More importantly, our model outperforms GLMP by 2.0% overall, 3.3% in the Navigate domain, 1.1% in the Weather domain and 0.6% in Schedule domain on entity F1 metric, which indicates that considering relevance between target domain input and all domains is effective for enhancing performance of each domain. On Multi-Woz 2.1 dataset, the same trend of improvement has been witnessed, which further shows the effectiveness of our framework. We study the strengths of our model from several perspectives on SMD dataset. We first conduct several ablation experiments to analyze the effect of different components in our framework. Next, we conduct domain adaption experiments to verify the transferability of our framework given a new domain with little or no labeled data. In addition, we provide a visualization of the dynamic fusion layer and case study to better understand how the module affects and contributes to the performance. Several ablation experiments and results are shown in Table 3. In detail, 1) w/o Domain-shared Knowledge Transfer denotes that we remove domain-shared feature and just keep fused domain-specific feature for generation. 2) w/o Domain Fusion mechanism denotes that we simply sum all domain-specific features rather than use the MOE mechanism to dynamically fuse domain-specific features. 3) w/o Multi-Encoder represents that we remove multi-encoder module and adopt one shared encoder in our framework. 4) w/o Multi-Decoder represents that we remove the multi-decoder module and adopt one shared decoder. 5) w/o Adversarial Training denotes that we remove the adversarial training in experimental setting. Generally, all the proposed components are effective to contribute the final performance. Specifically, we can clearly observe the effectiveness of our dynamic fusion mechanism where w/o domain-specific knowledge fusion causes 1.8% drops and the same trend in removing domain-shared knowledge fusion. This Figure 7: Distribution of Mix-of-the-expert mechanism across source domains for randomly selected 100 examples in each domain on SMD dataset. further verifies that domain-shared and domain-specific feature are benefit for each domain performance. Low-Resource Setting To simulate low-resource setting, we keep two domains unchanged, and the ratio of the except domain from original data varies from [1%, 5%, 10%, 20%, 30%, 50%]. The results are shown in Figure 5. We can find that: (1) Our framework outperforms the GLMP baseline on all ratios of the original dataset. When the data is only 5% of original dataset, our framework outperforms GLMP by 13.9% on all domains averagely. (2) Our framework trained with 5% training dataset can achieve comparable and even better performance compared to GLMP with 50% training dataset on some domains. This implies that our framework effectively transfers knowledge from other domains to achieve better performance for the low-resources new domain. Zero-Shot Setting Specially, we further evaluate the performance of domain adaption ability on the zero-shot setting given an unseen domain. We randomly remove one domain from the training set, and other domain data remained unchanged to train the model. During test, the unseen domain input use the MoE to automatically calculate the correlation between other domains and the current input and get the results. Results are shown in 0.1996 0.0242 0.7762 0 1 Navigation Weather Schedule 0.0015 0.1457 0.8527 0 1 Navigation Weather Schedule 0.0249 0.8908 0.0843 0 1 Navigation Weather Schedule Driver :Manhattan,please will it be cloudy on Monday ? 0.0015 0.1457 0.8527 0 0.5 1 Navigation Weather Schedule Driver :Manhattan,please will it be cloudy on Monday ? Car :Mondaywillbefoggy. Weather@5% 0.1996 0.0242 0.7762 0 0.5 1 Navigation Weather Schedule DialogueHistory :Find location and address to home that is nearest me.Your home is at 56cadwellstreet.Thanks, set the gps for there. Figure 8: Case of of expert gate distribution in SMD dataset. Text segments with red color represents appearing in both schedule and navigation domain. Figure 6, we can see our model significantly outperforms GLMP on three domains, which further demonstrate the transferability of our framework. To better understand what our dynamic fusion layer has learned, we visualize the gate distribution for each domain in low-resource (5%) setting, which fuses domain-specific knowledge among various cases. As shown in the Figure 7, for a specific target domain, different examples may have different gate distributions, which indicates that our framework successfully learns how to transfer knowledge between different domains. For example, the navigation column contains 100 examples from its test set and each row show the corresponding expert value. More specifically, in the navigation column, we observe that the expert value in schedule domain is bigger than weather domain, which indicates schedule domain transfers more knowledge to navigation than weather domain. Furthermore, we provide one case for navigation domain and their corresponding expert gate distribution. The cases are generated with 5% training data in the navigation domain and other two domain datasets keep the same, which can better show how the other two domains transfer knowledge to the low-resource domain. As shown in Figure 8, the expert value of schedule domain is bigger than the weather domain, which indicates the schedule contributes more than weather domain. In further exploration, we find word  location  and  set  appear both in navigation and schedule domain, which shows schedule has closer relation with navigation than weather, which indicates our model successfully transfers knowledge from the closest domain. We provide human evaluation on our framework and other baseline models. We randomly generated 100 responses. These responses are based on distinct dialogue history on the SMD test data. Following Wen et al. (2018) and Qin et al. (2019b), We hired human experts and asked them to judge the quality of the responses according to correctness, fluency, and humanlikeness on a scale from 1 to 5. Results are illustrated in Table 4. We can see that our framework outperforms GLMP on all metrics, which is consistent with the automatic evaluation. Existing end-to-end task-oriented systems can be classified into two main classes. A series of work trains a single model on the mixed multi-domain dataset. Eric et al. (2017) augments the vocabulary distribution by concatenating KB attention to generatge entities. Lei et al. (2018) first integrates track dialogue believes in end-to-end task-oriented dialog. Madotto et al. (2018) combines end-to-end memory network (Sukhbaatar et al., 2015) into sequence generation. Gangi Reddy et al. (2019) proposes a multi-level memory architecture which first addresses queries, followed by results and finally each key-value pair within a result. Wu et al. (2019a) proposes a global-to-locally pointer mechanism to query the knowledge base. Compared with their models, our framework can not only explicitly utilize domain-specific knowledge but also consider different relevance between each domain. Another series of work trains a model on each domain separately. Wen et al. (2018) leverages dialogue state representation to retrieve the KB implicitly. Qin et al. (2019b) first adopts the KB-retriever to explicitly query the knowledge base. Their works consider only domain-specific features. In contrast, our framework explicitly leverages domain-shared features across domains. The shared-private framework has been explored in many other task-oriented dialog components. Liu and Lane (2017) applies a shared-private LSTM to generate shared and domain-specific features. Zhong et al. (2018) proposes a global-local architecture to learn shared feature across all slots and specific feature for each slot. More recently, Zhang et al. (2018) utilizes the shared-private model for text style adaption. In our work, we explore shared-private framework in end-to-end task-oriented dialog to better transfer domain knowledge for querying knowledge base. In addition, we take inspiration from Guo et al. (2018), who successfully apply the mix-of-the-experts (MoE) mechanism in multi-sources domain and cross-lingual adaption tasks. Our model not only combines the strengths of MoE to incorporate domain-specific feature, but also applies adversarial training to encourage generating shared feature. To our best of knowledge, we are the first to effectively explore shared-private framework in multi-domain end-to-end task oriented dialog. In this paper, we propose to use a shared-private model to investigate explicit modeling domain knowledge for multi-domain dialog. In addition, a dynamic fusion layer is proposed to dynamically capture the correlation between a target domain and all source domains. Experiments on two datasets show the effectiveness of the proposed models. Besides, our model can quickly adapt to a new domain with little annotated data. We thank Min Xu, Jiapeng Li, Jieru Lin and Zhouyang Li for their insightful discussions. We also thank all anonymous reviewers for their constructive comments. This work was supported by the National Natural Science Foundation of China (NSFC) via grant 61976072, 61632011 and 61772153. Besides, this work also faxed the support via Westlake-BrightDreams Robotics research grant.
abstain abstain abstain abstain objective objective result result abstain other abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain objective method result result objective objective other abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain other abstain method method method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method other abstain abstain abstain method abstain method abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain other abstain result abstain result result result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain result abstain abstain abstain result other other other other other other other method other other other other objective other other other other objective method method objective objective abstain abstain objective other other other other