Existing end-to-end dialog systems perform less effectively when data is scarce. To obtain an acceptable success in real-life online services with only a handful of training examples, both fast adaptability and reliable performance are highly desirable for dialog systems. In this paper, we propose the Meta-Dialog System (MDS), which combines the advantages of both meta-learning approaches and human-machine collaboration. We evaluate our methods on a new extended-bAbI dataset and a transformed MultiWOZ dataset for low-resource goal-oriented dialog learning. Experimental results show that MDS significantly outperforms non-meta-learning baselines and can achieve more than 90% per-turn accuracies with only 10 dialogs on the extended-bAbI dataset. End-to-end neural models have shown a great potential in building flexible goal-oriented dialog systems. They can be directly trained on past dialogs without any domain-specific handcrafting, which makes it easy to automatically scale up to new domains (Bordes et al., 2017). However, these models are normally data-hungry and have only been successfully applied to domains with rich datasets (Perez et al., 2017; Luo et al., 2019; Kim et al., 2019). In real-world scenarios, common issues with end-to-end dialog models include: (1) the shortage of proper training dialogs because of the high cost of data collection and cleaning, i.e., the data scarcity problem (Zhao and Eskenazi, 2018), and (2) a large gap between limited data and unknown online test examples, i.e., the covariate shift effect (Liu et al.). Such problems can lead to a significant performance degradation in dialog systems, which  Corresponding author may harm the users' experience and result in loss of customers in commercial applications. Therefore, both fast adaptability and reliable performance are strongly desirable for practical system deployment. Fast adaptability reflects the efficiency of adapting dialog systems to domains with low-resource data. Reliable performance reflects the robustness of handling unpredictable user behaviors in online services. To boost the online performance of dialog systems, there have been some recent work (Rajendran et al., 2019; Wang et al., 2019; Lu et al., 2019) on designing end-to-end models in a human-machine joint-teaming manner. For instance, the dialog system in (Rajendran et al., 2019) can identify an ongoing dialog during testing when the system might fail and transfer it to a human agent. But all these methods are trained with sufficient data, which hinders the possibility of rapidly prototyping the models in new domains with restricted resources. In this paper, we formulate the low-resource goal-oriented dialog learning as a few-shot learning problem, where a limited numbers of dialogs are used for training and the remaining for the test. We propose the Meta-Dialog System (MDS), an end-to-end human-machine teaming framework optimized by the model-agnostic meta-learning (MAML) algorithm (Finn et al., 2017). In general, MDS learns to make prediction and requests human by finding good initial parameters, which can be adapted to new tasks fast and reliably by using fewer dialogs. We evaluate our methods on a new multi-domain dialog dataset called extended-bAbI . Results show that MDS achieves obvious performance improvement over baselines and attains more than 90% per-turn accuracy on new domains with only 10 dialogs. We also perform experiments on MultiWOZ dataset (Eric et al., 2019) which has been transformed into simplified bAbI format and observe similar superior results with MDS. In summary, the main contributions of this paper are three-fold: (1) To the best of our knowledge, this is the first study on applying meta-learning to retrieval-based end-to-end goal-oriented dialog systems; (2) we leverage the MAML algorithm to optimize a human-machine collaborative dialog system and show very promising results on the low-resource dialog tasks; and (3) we propose a new dataset and hope that can help bring forward the research in this area. In this section, we first introduce the problem definition and our new dataset; we then elaborate the framework of MDS and meta-learning procedures. Problem Definition. We focus on the retrieval-based goal-oriented dialog tasks (Perez et al., 2017), where a training data d i usually contains a triple ( H i , y i , R ) . H i denotes the dialog history consisting of all user utterances and system responses up to the current turn, R is a set of given candidate responses and y i is the index of the correct response in R . The main task is to train an end-to-end dialog model to predict y i from R based on H i . Extended-bAbI Dataset. The original bAbI dataset (Bordes et al., 2017) is not suitable for low-resource settings due to the lack of domains and tasks. We extend it into a multi-domain dataset through complicated simulation rules and construct templates with a more diversity to raise the diffi-culty. There are 7 domains in total: restaurant, flights, hotels, movies, music, tourism and weather , each of which has its own ontology and the candidate response set. Similar to (Bordes et al., 2017), a complete dialog in extended-bAbI contains four phases of interactions: (1) the system asks for required attributes to constrain the search and issues the first API call; (2) the user updates their requests for revised API calls; (3) the system con-firms for multiple times to determine the entity the user wants; (4) the user requests more attributes for extra information based on the final entity. The total number of dialogs is 21,000 and the detailed examples and statistics are given in Appendix A.1. In MDS, there is an encoding module to extract neural features of dialogs and a policy module to make system actions of either predicting responses or requesting human. All modules are jointly optimized with the MAML algorithm. The main framework of training MDS is shown in Figure 1. Encoding Module. It contains a history encoder to compute the dialog state vector s i for H i and a response encoder to compute the response embedding r j for the j -th response in R . The dimensions of s i and r j are set as the same. In this paper, we use the MemN2N (Sukhbaatar et al., 2015) as the history encoder and a simple additive model for the response encoder, but many other models optimized by gradient descent may be applied here. Policy Module. This module consists of a switch S that makes a binary decision whether to request human to select the response, and a response predictor P that predicts the right response itself if human is not requested. We assume that the response chosen by human is always correct. For the optimization of P , the widely used large-margin cosine loss (Wang et al., 2018; Lin and Xu, 2019) is employed since it maximizes the decision margin in the angular space and is able to force the model to learn more discriminative deep features. Suppose a batch of training data is D = { d 1 , ...d i , ..., d |D| } , then the formulation is: LLMC = |D| (cid:88) i =1  log e a  (cos( s i ,r yi )  b ) e a  (cos( s i ,r yi )  b ) + (cid:80) j (cid:54) = y i e a  cos( s i ,r j ) (1) where cos(  ,  ) is a function that calculates the cosine similarity of two input vectors. a is the scaling factor and b is the cosine margin ( a = 30 , m = 0 . 1 in our experiments). In the test phase, the model predicts an answer according to the maximal cosine angle y  i = argmax j cos( s i , r j ) . The switch S is a neural binary classifier that also takes s i and each r j as input and calculate the decision probability of requesting human as follows: w ij = e s T i Wr j / (cid:88) |R| k =1 e s T i Wr k (2) c i = (cid:88) |R| j =1 w ij r j (3) f i = s i  c i (4) p i =  ( FC ( f i )) (5) where  is the sigmoid function and  the concatenation function for vectors. FC(  ) is a fully-connected neural network with one hidden layer that has half size of the input layer and is activated Figure 1: An overview of training the Meta-Dialog System. by tanh function. |R| is the size of R and W is a trainable square matrix. Learning to switch. Since there are no actual labels for S to indicate whether it is correct to ask human or not, some previous work (Woodward and Finn, 2016; Rajendran et al., 2019) proposes to use the REINFORCE algorithm (Williams, 1992) for weakly-supervised training, but their reward settings fail to penalize the case when the model asks human while it can give right prediction, which may lead to redundant requests. To consider this effect, we propose a new reward definition here. For the batch data D , we calculate the F1 scores 1 for positive data and negative data, respectively, and take the average of them to get a scalar value score ( D ) . Then each data d i  D is assigned with a reward by computing an incremental value as below: r t = score ( D )  score ( D  d i ) (6) Through maximizing such rewards, the switch S learns to be more effective and asks human when it is necessary. The reinforcement learning loss for S is LRL = (cid:80) |D| i =1  r i log p i , and the final loss of our model is L = LLMC + LRL . We rewrite the final loss L as L ( M  , D ) for clarity, where M  denotes the dialog model with trainable parameters  and D is the batch data for training. During meta-learning, we first choose one domain as the target domain and the rest as source domains. Then we uniformly sample K different domains T = {  1 , . . . ,  K } from source domains as meta-tasks. For each meta-task  k , we sample N data as the support set D sup k and other N data with the same answers as the query set D que k . 1 Detailed explanations can be found in Appendix A.2. Algorithm 1 Meta-learning for MDS Input: The learning rates  ,  Output: optimal meta-learned model 1: Initialize model parameters  randomly 2: while not converged do 3: Sample T from source domains and prepare D sup k , D que k 4: for each  k do 5: Evaluate L ( M  , D sup k ) 6: Compute  (cid:48) k =      L ( M  , D sup k ) 7: Evaluate L ( M  (cid:48) k , D que k ) 8: end for 9: Update        (cid:80) Kk =1 L ( M  (cid:48) k , D que k ) 10: end while M  is first updated on support sets for each  k :  (cid:48) k =      L ( M  , D sup k ) (7) Then M  is evaluated on each D que k with  (cid:48) k respectively and is optimized as follows:        (cid:88) K k =1 L ( M  (cid:48) k , D que k ) (8) where ,  are learning rates. By training on multiple tasks via MAML, M  can learn good initial parameters that is applicable on new tasks or domains (Finn et al., 2017; Mi et al., 2019). The algorithm is summarised in Algorithm 1. After this meta-learning as pre-training, we fine-tune M  on the target domain with the first L dialogs of its training set, where L is a small number. To mimic the situation of online testing, we evaluate M  on the whole test sets and regard those unseen user utterances as new user behaviours. In our experiments, we first verify the capability of MDS on our newly simulated dialog dataset extended-bAbI , and then conduct extra evaluation on the more realistic dataset MultiWOZ 2.1 (Eric et al., 2019). We select each domain as the target domain in turn and take the average of the results in all domains. Metric. Following (Wang et al., 2019), we report the user-perceived per-turn accuracy (per-turn accuracy' is used in the remainder of the paper), where the prediction of one turn is considered correct if the model either selects the right response by itself or asks human. To be fair, we also report the human request rate. The less the request rate and higher per-turn accuracy are, the more reliable the model performs online. Implementation details. For the meta-learning, we use SGD for the inner loop and Adam for the outer loop with learning rate  = 0 . 01 and  = 0 . 001 . The meta-task size K is 4 and the support or query set size N is 16. For the standard MLE training, we use Adam with a learning rate of 0.001 and set the batch size as 32. Both schemes are trained for a maximum of 5000 iterations with early stopping on the validation set. During fine-tuning on new domains, we use SGD with the learning rate 0.01 for all models and report the final results after fine-tuning 10 iterations on L training dialogs of the target domain, where L = 0 , 1 , 5 , 10 . The word vector size is 25 and all MemN2Ns take 3 hops. We compare MDS with the following baselines:  Mem : A MemN2N (Sukhbaatar et al., 2015) model trained with standard MLE. MetaMem : A MemN2N trained with MAML. Both Mem and MetaMem can not request human. Mem+C : A MemN2N model combined with a binary classifier in (Rajendran et al., 2019), which has different objective functions and optimization. IDS : The incremental dialog system used in (Wang et al., 2019), which requests human by estimating the uncertainty through a variational autoencoder. MDS -switch : A MDS without the switch S . MDS rand : A MDS whose switch is replaced with a random classifier that has the same request rate. MDS mle : A MDS whose meta-learning optimization is replaced with standard MLE. Table 1 shows few-shot adaptation results for different methods. MDS significantly outperforms other models under all adaptation sizes of new dialogs and can achieve a 91.31% per-turn accuracy on average with only 10 new dialogs. There is a gap between methods without the switch (such as Mem, MetaMem and MDS -switch ) and methods with the switch in Table 1, indicating that the switch S is crucial for improving the overall per-turn accuracy because of the human agent. However, without proper objective functions and meta-learning optimization, Mem+C and IDS 2 have poorer performances in both metrics than MDS even if they contain the switch module. In the ablation study, we see a steady increase of about 10% per-turn accuracy from the comparison between MDS and MDS rand , suggesting that the switch does identify intractable dialogs. MDS mle is the closest baseline to MDS, but we still observe an obvious improvement, which means joint optimization of S and P via meta-learning allows faster and better adaptation while maintaining similar request rates. Appendix A.3 illustrates detailed case studies for different methods. To further investigate the adaptation process, we present the fine-tuning curves for different methods with 1 dialog adaptation in Figure 2. As it can be seen, MDS achieves the best accuracy at the beginning and converges fastest as well, showing that it can transfer on new tasks quickly by finding better parameter initialization. MultiWOZ (Budzianowski et al., 2018) is a widely-used multi-domain Wizard-of-Oz dialog dataset spanning 7 distinct domains and containing 10k dialogs. This realistic dataset has been a standard benchmark for various dialog tasks such as belief tracking and policy optimization. In our experiment, we use the corrected version MultiWOZ 2.1 (Eric et al., 2019) for evaluation. To translate the MultiWOZ dialogs into bAbI-format data, we first delexicalize the slot-values in user utterances using dialog labels, and then produce a set of canonical system acts as the candidate responses by simplifying the original dialog acts. Only dialogs containing single domain are used in our experiments and a MultiWOZ dialog sample is given in Appendix A.4. Table 2 shows the adaptation results for different models on MultiWOZ 2.1. It can be seen that MDS still largely outperforms other models with the adaptation of 10 dialogs. The degradation of per-turn accuracy from extended-bAbI to MultiWOZ is reasonable since the user utterance is more diverse and the dialog policy is more flexible. End-to-end neural approaches of building dialog systems have attracted increasing research interest. The work of (Bordes et al., 2017) is the first attempt to solve goal-oriented dialog tasks with end-to-end models. Further improvements has been made in (Williams et al., 2017) to combine explicit domain-specific knowledge and implicit RNN features. Luo et al. (2019) take user personalities into consideration for better user satisfaction. Rajendran et al. (2018) learn dialogs with multiple possible answers. Our work is inspired by the work of (Rajendran et al., 2019; Wang et al., 2019), which Method Adapt with 10 dialogs accuracy request Mem 56.87  1.63 n.a. MetaMem 62.78  2.05 n.a. Mem+C 80.59  3.13 38.18  5.01 MDS -switch 64.50  3.75 n.a. MDS rand 74.78  4.35 38.34 MDS mle 80.92  3.02 37.91  4.20 MDS 83.52  3.30 38.34  6.96 Table 2: Few-shot test results on MultiWOZ 2.1. propose to solve unseen user behaviors through human-machine teamwork. The research of (Liu et al.; Chen et al., 2017; Lu et al., 2019) also show the advantages of incorporating the role of human to teach online. However, dialog learning in low-resource scenarios has not been investigated. Meta-learning aims to learn new tasks rapidly with a few training examples (Sung et al., 2018; Finn et al., 2017), which fits well to our task. There have been some work applying meta-learning to other tasks in dialog research, such as that in (Dou et al., 2019; Geng et al., 2019) for natural language understanding and (Qian and Yu, 2019; Mi et al., 2019) for natural language generation. In this paper, we leverage the MAML algorithm to optimize a human-machine collaborative dialog system, which shows good results for both fast adaptability and reliable performance. In the future, we plan to use more powerful encoders and evaluate our methods on real dialog data. The research of the last author is supported by the Natural Sciences and Engineering Research Council of Canada (NSERC).
abstain abstain objective objective abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain objective abstain abstain objective abstain result other objective abstain abstain abstain abstain abstain abstain objective abstain other abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain other abstain abstain other abstain abstain abstain abstain other abstain method abstain abstain abstain abstain method method abstain other abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain method abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other other other other other other abstain other result method other