Generating fluent and informative responses is of critical importance for task-oriented dialogue systems. Existing pipeline approaches generally predict multiple dialogue acts first and use them to assist response generation. There are at least two shortcomings with such approaches. First, the inherent structures of multi-domain dialogue acts are neglected. Second, the semantic associations between acts and responses are not taken into account for response generation. To address these issues, we propose a neural co-generation model that generates dialogue acts and responses concurrently. Unlike those pipeline approaches, our act generation module preserves the semantic structures of multi-domain dialogue acts and our response generation module dynamically attends to different acts as needed. We train the two modules jointly using an uncertainty loss to adjust their task weights adaptively. Extensive experiments are conducted on the large-scale MultiWOZ dataset and the results show that our model achieves very favorable improvement over several state-of-the-art models in both automatic and human evaluations. Task-oriented dialogue systems aim to facilitate people with such services as hotel reservation and ticket booking through natural language conversations. Recent years have seen a rapid proliferation of interests in this task from both academia and industry (Bordes et al., 2017; Budzianowski et al., 2018; Wu et al., 2019). A standard architecture of these systems generally decomposes this task into several subtasks, including natural language understanding (Gupta et al., 2018), dialogue state tracking (Zhong et al., 2018) and natural language  Xiaojun Quan is the corresponding author of this paper. Most of this work was done when Kai Wang was working as an intern at Alibaba DAMO Academy. generation (Su et al., 2018). They can be modeled separately and combined into a pipeline system. Figure 1 shows a dialogue example, from which we can notice that the natural language generation subtask can be further divided into dialogue act prediction and response generation (Chen et al., 2019; Zhao et al., 2019; Wen et al., 2017). While the former is intended to predict the next action(s) based on current conversational state and database information, response generation is used to produce a natural language response based on the action(s). In order for dialogues to be natural and effective, responses should be fluent, informative, and relevant. Nevertheless, current sequence-to-sequence models often generate uninformative responses like I don't know (Li et al., 2016a), hindering the dialogues to continue or even leading to a failure. Some researchers (Pei et al., 2019; Mehri et al., area root hotel restaurant attraction name request inform phone domain action slot Dialog Act Graph Sequence Generation (Ours) Multiple Binary Classification (HDSA) hotel ... inform...phone reference < sos > rest. 2019) sought to combine multiple decoders into a stronger one to avoid such responses, while others (Chen et al., 2019; Wen et al., 2015; Zhao et al., 2019; Wen et al., 2017) represent dialogue acts in a global, static vector to assist response generation. As pointed out by Chen et al. (2019), dialogue acts can be naturally organized in hierarchical structures, which has yet to be explored seriously. Take two acts station-request-stars and restaurant-inform-address as an example. While the first act rarely appears in real-world dialogues, the second is more often. Moreover, there can be multiple dialogue acts mentioned in a single dialogue turn, which requires the model to attend to different acts for different sub-sequences. Thus, a global vector is unable to capture the inter-relationships among acts, nor is it flexible for response generation especially when more than one act is mentioned. To overcome the above issues, we treat dialogue act prediction as another sequence generation problem like response generation and propose a co-generation model to generate them concurrently. Unlike those classification approaches, act sequence generation not only preserves the interrelationships among dialogue acts but also allows close interactions with response generation. By attending to different acts, the response generation module can dynamically capture salient acts and produce higher-quality responses. Figure 2 demonstrates the difference between the classification and the generation approaches for act prediction. As for training, most joint learning models rely on hand-crafted or tunable weights on development sets (Liu and Lane, 2017; Mrksic et al., 2017; Rastogi et al., 2018). The challenge here is to combine two sequence generators with varied vocabularies and sequence lengths. The model is sensitive during training and nontrivial to generate an optimal weight. To address this issue, we opt for an uncertainty loss (Kendall et al., 2018) to adaptively adjust the weight according to task-specific uncertainty. We conduct extensive studies on a large-scale task-oriented dataset to evaluate the model. The experimental results confirm the effectiveness of our model with very favorable performance over several state-of-the-art methods. The contributions of this work include:  We model dialogue act prediction as a sequence generation problem that allows to exploit act structures for the prediction. We propose a co-generation model to generate act and response sequences jointly, with an uncertainty loss used for adaptive weighting. Experiments on MultiWOZ verify that our model outperforms several state-of-the-art methods in automatic and human evaluations. Dialogue act prediction and response generation are closely related in general in the research of dialogue systems (Chen et al., 2019; Zhao et al., 2019; Wen et al., 2017), where dialogue act prediction is first conducted and used for response generation. Each dialogue act can be treated as a triple (domain-action-slot) and all acts together are represented in a one-hot vector (Wen et al., 2015; Budzianowski et al., 2018). Such sparse representation makes the act space very large. To overcome this issue, Chen et al. (2019) took into account act structures and proposed to represent the dialogue acts with level-specific one-hot vectors. Each dimension of the vectors is predicted by a binary classifier. To improve response generation, Pei et al. (2019) proposed to learn different expert decoders for different domains and acts, and combined them with a chair decoder. Mehri et al. (2019) applied a cold-fusion method (Sriram et al., 2018) to combine their response decoder with a language model. Zhao et al. (2019) treated dialogue acts as latent variables and used reinforcement learning to optimize them. Reinforcement learning was also applied to find optimal dialogue policies in task-oriented dialogue systems (Su et al., 2017; 7127 Williams et al., 2017) or obtain higher dialog-level rewards in chatting (Li et al., 2016b; Serban et al., 2017). Besides, Chen et al. (2019) proposed to predict the acts explicitly with a compact act graph representation and employed hierarchical disentangled self-attention to control response text generation. Unlike those pipeline architectures, joint learning approaches try to explore the interactions between act prediction and response generation. A large body of research in this direction uses a shared user utterance encoder and train natural language understanding jointly with dialogue state tracking (Mrksic et al., 2017; Rastogi et al., 2018). Liu and Lane (2017) proposed to train a unified network for two subtasks of dialogue state tracking, i.e., knowledge base operation and response candidate selection. Jiang et al. (2019) showed that joint learning of dialogue act and response benefits representation learning. These works generally demonstrate that joint learning of the subtasks of dialogue systems is able to improve each other and the overall system performance. Let T = { U 1 , R 1 , . . . , U t  1 , R t  1 , U t } denote the dialogue history in a multi-turn conversational setting, where U i and R i are the i -th user utterance and system response, respectively. D = { d 1 , d 2 , . . . , d n } includes the attributes of related database records for current turn. The objective of a dialogue system is to generate a natural language response R t = y 1 y 2 . . . y n of n words based on the current belief state and database attributes. In our framework, dialogue acts and response are co-generated based on the transformer encoder-decoder architecture (Vaswani et al., 2017). A standard transformer includes a multi-head attention layer that encodes a value V according to the attention weights from query Q to key K , followed by a position-wise feed-forward network ( G f ): O = V + G f ( MultiHead ( Q, K, V )) (1) where Q, K, V, O  R n  d . Encoder We use E = Emb ([ T ; D ]) to represent the concatenated word embeddings of dialogue history T and database attributes D . The transformer F ( Q, K, V ) is then used to encode E and output its hidden state H e : H e = F ( E, E, E ) (2) Decoder At each time step t of response generation, the decoder first computes a self-attention h rt over already-generated words y 1: t  1 : h rt = F ( e rt  1 , e r 1: t  1 , e r 1: t  1 ) (3) where e rt  1 is the embedding of the ( t  1) -th generated word and e r 1: t  1 is an embedding matrix of e r 1 to e rt  1 . Cross-attention from h rt to dialogue history T is then executed: c rt = F ( h rt , H e , H e ) (4) The resulting vectors of Equations 3 and 4, h rt and c rt , are concatenated and mapped to a distribution of vocabulary size to predict next word: p ( y t | y 1: t  1 ) = softmax ( W r [ c rt ; h rt ]) (5) 4 The MARCO Approach Based on the above encoder-decoder architecture, our model is designed to consist of three components, namely, a shared encoder, a dialogue act generator, and a response generator. As shown in Figure 3, instead of predicting each act token individually and separately from response generation, our model aims to generate act sequence and response concurrently in a joint model which is optimized by the uncertainty loss (Kendall et al., 2018). Dialogue acts can be viewed as a semantic plan for response generation. As shown in Figure 2, they can be naturally organized in hierarchical structures, including domain level, action level, and slot level. Most existing methods treat dialogue acts as triples represented in one-hot vectors and predict the vector values with binary classifiers (Wen et al., 2015; Budzianowski et al., 2018). Such representations ignore the inter-relationships and associations among acts, domains, actions and slots. For example, the slot area may appear in more than one domain. Unlike them, we model the prediction of acts as a sequence generation problem, which takes into consideration the structures of acts and generates each act token conditioned on its previously-generated tokens. In this approach, different domains are allowed to share common slots and the search space of dialogue act is greatly reduced. sequence is organized by domain, action and slot, while items at each level are arranged in dictionary order, where identical items are merged. When decoding each act token, we first represent the current belief state with an embedding vector v b and add it to each act word embedding e at as: u at = W b v b + e at . Finally, the decoder of Section 3.2 is used to generate hidden states H a and act tokens accordingly. Dialogue acts and responses are closely related in dialogue systems. On one hand, system responses are generated based on dialogue acts. On the other, their shared information can improve each other through joint learning. Shared Encoder Our dialogue act generator and response generator share one same encoder and input, but having different masking strategies for the input to focus on different information. In particular, only the current utterance is kept for act generation, while the entire history utterances are used for response generation. 1 1 Empirical evidences show that act generation is more related to the current utterance, while response generation benefits more from long dialogue history. Dynamic Act Attention A response usually corresponds to more than one dialogue act in multi-domain dialogue systems. Nevertheless, existing methods mostly use a static act vector to represent all the acts, and add the vector to each response token representation. They ignore the fact that different subsequences of a response may need to attend to different acts. To address this issue, we compute dynamic act attention o rt from the response to acts when generating a response word: o rt = F ( h rt , H a , H a ) (7) where h rt is the current hidden state produced by Equation 3. Then, we combine o rt and h rt with response-to-history attention c rt (by Equation 4) to estimate the probabilities of next word: p ( y t | y 1: t  1 ) = softmax ( W r [ h rt ; c rt ; o rt ]) (8) Uncertainty Loss The cross-entropy function is used to measure the generation losses, L a (  ) and L r (  ) , of dialogue acts and responses, respectively: L a (  ) =  T a (cid:2) j =1 log p ( a  ( i ) j | a ( i ) 1: j  1 , T, D, v b ) (9) L r (  ) =  T r (cid:2) j =1 log p ( y  ( i ) j | y ( i ) 1: j  1 , T, D, A ) (10) 7129 where the ground-truth tokens of acts and response of each turn are represented by A  and Y  , while the predicted tokens by A and Y . To optimize the above functions jointly, a general approach is to compute a weighted sum like: L (  ) =  L a (  ) + (1   ) L r (  ) (11) However, dialogue acts and responses vary seriously in sequence length and vocabulary size, making the weight  unstable to tune. Instead, we opt for an uncertainty loss (Kendall et al., 2018) to adjust it adaptively: L ( ,  1 ,  2 ) = 1 2  21 L a (  )+ 1 2  22 L r (  )+log  21  22 (12) where  1 and  2 are two learnable parameters. The advantage of this uncertainty loss is that it models the homoscedastic uncertainty of each task and provides task-dependent weight for multi-task learning (Kendall et al., 2018). Our experiments also confirm that it leads to more stable weighting than the traditional approach (Section 6.3). MultiWOZ 2.0 (Budzianowski et al., 2018) is a large-scale multi-domain conversational datatset consisting of thousands of dialogues in seven domains. For fair comparison, we use the same validation set and test set as previous studies (Chen et al., 2019; Zhao et al., 2019; Budzianowski et al., 2018), each set including 1000 dialogues. 2 We use the Inform Rate and Request Success metrics to evaluate dialog completion, with one measuring whether a system has provided an appropriate entity and the other assessing if it has answered all requested attributes. Besides, we use BLEU (Papineni et al., 2002) to measure the fluency of generated responses. To measure the overall system performance, we compute a combined score: ( Inform Rate + Request Success )  0 . 5 + BLEU as before (Budzianowski et al., 2018; Mehri et al., 2019; Pei et al., 2019). The implementation 3 is on a single Tesla P100 GPU with a batch size of 512. The dimension of 2 There are only five domains ( restaurant , hotel , attract , taxi , train ) of dialogues in the test set as the other two ( hospital , police ) have insufficient dialogues. word embeddings and hidden size are both set to 128. We use a 3-layer transformer with 4 heads for the multi-head attention layer. For decoding, we use a beam size of 2 to search for optimal results, and apply trigram avoidance (Paulus et al., 2018) to fight trigram-level repetition. During training, we first train the act generator for 10 epochs for warmup and then optimize the uncertainty loss with the Adam optimizer (Kingma and Ba, 2015). A few mainstream models are used as baselines for comparison with our neural co-generation model (MARCO ), being categorized into three categories: Without Act . Models in this category directly generate responses without act prediction, including LSTM (Budzianowski et al., 2018), Transformer (Vaswani et al., 2017), TokenMoE (Pei et al., 2019) and Structured Fusion (Mehri et al., 2019). One-Hot Act . In SC-LSTM (Wen et al., 2015), dialogue acts are treated as triples and information flow from acts to response generation is controlled by gates. HDSA (Chen et al., 2019) is a strong two-stage model, which relies on BERT (Devlin et al., 2019) to predict a one-hot act vector for response generation. Sequential Act . Since our model does not rely on BERT, to make a fair comparison with HDSA, we design the experiments from two aspects to ensure they have the same dialogue act inputs for response generation. First, the act sequences produced by our co-generation model are converted into one-hot vectors and fed to HDSA. Second, the predicted one-hot act vectors by BERT are transformed into act sequences and passed to our model as inputs. The overall results are shown in Table 1, in which HDSA (MARCO ) means HDSA using MARCO 's dialogue act information, and MARCO (BERT) means MARCO based on BERT's act prediction. From the table we can notice that our co-generation model (MARCO ) outperforms all the baselines in Inform Rate , Request Success , and especially in combined score which is an overall metric. By comparing the two HDSA models, we can find HDSA derives its main performance from the external BERT, which can also be used to improve our MARCO considerably (MARCO (BERT)). These 7130 Dialog Act Model Inform Success BLEU Combined Score Without Act LSTM 71.29 60.96 18.80 84.93 Transformer 71.10 59.90 19.10 84.60 TokenMoE 75.30 59.70 16.81 84.31 Structured Fusion 82.70 72.10 16.34 93.74 One-hot Act SC-LSTM 74.50 62.50 20.50 89.00 HDSA (MARCO ) 76.50 62.30 21.85 91.25 HDSA 82.90 68.90 23.60 99.50 Sequential Act MARCO 90.30 75.20 19.45 102.20 MARCO (BERT) 92.30 78.60 20.02 105.47 Table 1: Overall results on the MultiWOZ 2.0 dataset. results confirm the success of MARCO by modeling act prediction as a generation problem and training it jointly with response generation. Another observation is that despite its strong overall performance, MARCO shows inferior BLEU performance to the two HDSA models. The reason behind this is studied and analyzed in human evaluation (Section 7), showing that our model often generates responses inconsistent with references but favored by human judges. The performance of our model across different domains is also compared against HDSA. The average number of turns is 8.93 for single-domain dialogues and 15.39 for multi-domain dialogues (Budzianowski et al., 2018). As in Figure 4, our model shows superior performance to HDSA across all domains. The results suggest that MARCO is good at dealing with long dialogues. which is an updated version of MultiWOZ 2.0. As shown in Table 2, the overall results are consistent with that on MultiWOZ 2.0. More thorough studies and analysis are conducted in this section, trying to answer three questions: (1) How is the performance of our act generator in comparison with existing classification methods? (2) Can our joint model successfully build semantic associations between acts and responses? (3) How does the uncertainty loss contribute to our co-generation model? To evaluate the performance of our act generator, we compare it with several baseline methods mentioned in (Chen et al., 2019), including BiLSTM, Word-CNN, and 3-layer Transformer. We use MARCO to represent our act generator which is trained jointly with the response generator, and use Transformer (GEN) to denote our act generator without joint training. From Table 3, we notice that the separate generator, Transformer (GEN), performs much better than BiLSTM and Word-CNN, but comparable with Transformer. But after trained jointly with the response generator, MARCO manages to show the best performance, confirming the effect of the co-generation. To study the influence of the joint training and the dynamic act attention on response generation, we implement two pipeline approaches for comparison. We first train our act generator separately from response generation. Then, we keep its parameters fixed and train the response generator. The first baseline is created by replacing the dynamic act attention (Equation 7) with an average of the act hidden states, while the second baseline uses the dynamic act attention. As shown in Table 4, Pipeline 2 with dynamic act attention is superior to Pipeline 1 without it in all metrics, but inferior to the joint approach. Our joint model also surpasses the currently state-of-the-art pipeline system HDSA, even HDSA uses BERT. We find that by utilizing sequential acts, the dynamic act attention mechanism helps the response generator capture the local information by attending to different acts. An illustrative example is shown in Figure 5, where the response generator can attend to the local information such as day and stay as needed when generating a response asking about picking a different day or shorter stay. We reckon that by utilizing sequential acts, response generation benefits in two ways. First, the dynamic act attention allows the generator to attend to different acts when S e q u e n c i a l A c t Response Sequence Figure 5: An illustrative example of the dynamic act attention mechanism. generating a subsequence. Second, the joint training makes the two stages interact with each other, easing error propagation of pipeline systems. We opt for an uncertainty loss to optimize our joint model, rather than a traditional weighted-sum loss. To illustrate their difference, we conduct an experiment on the development set. For the traditional loss (Equation 11), we run for each weight from 0 to 1 stepped by 0.1. Note that since the weights,  1 and  2 , in the uncertainty loss are not hyperparam-eters but learned internally to each batch, we only record the best score within each round without giving the values of  1 and  2 . As shown in Figure 6, the uncertainty loss can learn adaptive weights with consistently superior performance. We conduct a human study to evaluate our model by crowd-sourcing. 4 For this purpose we randomly selected 100 sample dialogues (742 turns in total) from the test dataset and constructed two groups of systems for comparison: MARCO vs. HDSA and 4 The annotation results are available at https: //github.com/InitialBug/MarCo-Dialog/tree/master/human_evaluation 7132 MARCOHDSA MARCO vs. HDSA Readability Completion 17 2 74 94 9 4 Win Tie Lose MARCO Human Response MARCO vs. Human Response Readability Completion 30 9 47 75 23 16 100 100 100 100 Figure 7: Results of human study in response quality. MARCO vs. Human Response, where Human Response means the reference responses. Responses generated by each group were randomly assigned in pairs to 3 judges, who ranked them according to their completion and readability (Chen et al., 2019; Zhang et al., 2019). Completion measures if the response correctly answers a user query, including relevance and informativeness. Readability reflects how fluent, natural and consistent the response is. The results of this study are shown in Figure 7, where Win, Tie or Lose mean our MARCO system wins over, ties with or loses to its counterpart, respectively. From the results we note that MARCO outperforms HDSA and Human Response in completion, and ties 94% with HDSA in readability while underperforming Human Response. Overall speaking, MARCO is superior to HDSA and comparable with Human Response. We further analyzed the bad cases of our model in readability and found that our model slightly suffers from token level repetition, a problem that can be solved by methods like the coverage mechanism (Mi et al., 2016; Tu et al., 2016). In completion, our model can understand the users' need and tends to provides them more relevant information, so that they can finish their goals in shorter turns. We present two examples in Figure 8. In the first example, the user requests the hotel type while HDSA ignores it. The user requests to book one ticket in the second example, yet both HDSA and Human Response ask about the number once again. In contrast, our model directly answers the questions with correct information. To sum up, MARCO successfully improves the dialogue system by generating relevant and informative responses. In this paper, we presented a novel co-generation model for dialogue act prediction and response generation in task-oriented dialogue systems. Unlike previous approaches, we modeled act prediction as a sequence generation problem to exploit the semantic structures of acts and trained it jointly with response generation via dynamic attention from response generation to act prediction. To train this joint model, we applied an uncertainty loss for adaptive weighting of the two tasks. Extensive studies were conducted on a large-scale task-oriented dataset to evaluate the proposed model, and the results confirm its effectiveness with very favorable performance over several state-of-the-art methods. We thank the anonymous reviewers for their constructive reviews. This work was partially supported by the Fundamental Research Funds for the Central Universities (No.19lgpy220 and No.19lgpy219), the Program for Guangdong Introducing Innovative and Entrepreneurial Teams (No.2017ZT07X355), and the National Natural Science Foundation of China (No.61906217).
abstain abstain abstain abstain abstain objective abstain method result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain method result objective objective result other other other other other other other other other other other other other other other abstain abstain abstain method abstain method other other abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method other abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result other abstain abstain result abstain abstain result abstain abstain abstain method method abstain method method method abstain method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain method other abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain objective method method abstain other other