Typical generative dialogue models utilize the dialogue history to generate the response. However, since one dialogue utterance can often be appropriately answered by multiple distinct responses, generating a desired response solely based on the historical information is not easy. Intuitively, if the chatbot can foresee in advance what the user would talk about (i.e., the dialogue future) after receiving its response, it could possibly provide a more informative response. Accordingly, we propose a novel dialogue generation framework named ProphetChat that utilizes the simulated dialogue futures in the inference phase to enhance response generation. To enable the chatbot to foresee the dialogue future, we design a beam-search-like roll-out strategy for dialogue future simulation using a typical dialogue generation model and a dialogue selector. With the simulated futures, we then utilize the ensemble of a history-to-response generator and a future-to-response generator to jointly generate a more informative response. Experiments on two popular open-domain dialogue datasets demonstrate that ProphetChat can generate better responses over strong baselines, which validates the advantages of incorporating the simulated dialogue futures. Recent years have witnessed a surge of interest in building open-domain chatbots using generative approaches (Shang et al., 2015; Zhao et al., 2017; Tao et al., 2018; Zhang et al., 2020). These prevailing methods typically utilize dialogue histories as the dialogue context to generate the response via maximum likelihood estimation. Different from  Corresponding authors: Dongyan Zhao and Rui Yan. directed text generation tasks like machine translation where the target sentences are strictly constrained by the source sentence (Holtzman et al., 2019), the dialogue history and the response in chitchat conversations are loosely coupled (Feng et al., 2020a). In other words, open-domain chatbots often have more freedom" to decide what to respond since there often exists multiple distinct responses that can appropriately answer the given utterance. However, we argue that such excessive freedom" also reveals that the dialogue history only may not contain enough information to generate a desired response that is informative and easy to reply to. If provided with enriched dialogue contexts that contain more useful dialogue cues, it could be easier for the model to chat with a human. So here comes the questions: what kind of dialogue cues is complementary to dialogue histories, and how to obtain and use them. Recent studies in representation learning have demonstrated that when representing a token in a sentence, considering the tokens on its right side in addition to its left side can bring significant improvement (Devlin et al., 2019; Yang et al., 2019). Similar findings also appear in directed text generation where the future tokens on the right side can be beneficial to generate the current token (Serdyuk et al., 2017; Zhang et al., 2018c; Chen et al., 2020; Qi et al., 2020). Sharing the same spirit, we pursue to use the right side" information, which is the dialogue future in our task, as the complementary dialogue cue to enhance the generation of the current response. Intuitively, if the chatbot can be told in advance what the user would probably talk about (i.e., the dialogue future) after receiving its response, it only needs to provide a response that can smoothly connect the history and the future. To verify whether the dialogue future can act as the complementary dialogue cue, we conduct empirical studies. We find that using a dialogue generation model to learn the reverse dialogue flow (i.e., using the future to generate the response) is quite effective. Furthermore, when utilizing the ensemble of the history-to-response generation model and the future-to-response generation model to generate the response conditioned on both the history and the gold future, the quality of the generated response surely improves. Though effective, the ground truth dialogue future is inaccessible in the inference phase. Therefore, all existing works in this line choose to leverage the dialogue future only in the training phase (Shen et al., 2018; Feng et al., 2020a,b), leaving the inference phase unchanged. We argue that explicitly providing the possible dialogue futures in the inference phase can offer more direct help for the generation of the current response. To enable the incorporation of dialogue futures into response generation in the inference phase, we propose a response generation framework namely ProphetChat by answering two questions: how to acquire the future and how to use it. Figure 1 shows the generation process of ProphetChat. It consists of a history-to-response model (denoted as the forward model), a future-to-response model (denoted as the backward model), an ensemble gate, and a dialogue selector. Given a dialogue history, we first utilize an effective beam-search-like roll-out strategy to simulate possible dialogue futures. Concretely, the forward model first generates a batch of n possible responses based on the dialogue history. The dialogue selection model then comes to pick up the k -best responses. We further generate n possible futures for each of the picked responses, resulting in k  n futures. The selection model again picks up the k -best futures which are of higher quality compared with randomly sampled ones. Next, conditioned on both the history and the simulated future, we employ the forward model and the backward model to jointly generate the response by summing the per-step output probability distributions of the two models using a calculated weight. The weight is obtained by a trainable gate that learns to balance the trade-off between history and future information. Finally, we gather the k responses generated solely based on the history and the k  n responses generated based on both the history and the future, and use the selector to choose the top-ranked one as the final response. Since the ensemble generation model relies on the selector to sequentially select the response and the future, and the ensemble generation model also needs to learn how to balance the history and the future information given the selected future, we jointly train the whole model to make each module better collaborate with others to fulfill the ultimate goal: to maximize the likelihood of the gold response estimated by the ensemble generation model given the history and the selected future. We train the ensemble generation model directly using MLE objective while adopting reinforcement learning to tune the selector. Our contributions in this paper are three folds:  We propose a novel dialogue generation framework named ProphetChat which leverages the simulated dialogue future to enhance response generation through the ensemble of the history-to-response generator and the future-to-response generator. To the best of our knowledge, we are the first to utilize the dialogue futures for response generation in the inference phase.  To acquire better dialogue futures in the inference phase, we propose an effective beam-search-like roll-out strategy for dialogue future simulation with the help of a dialogue selector.  We conduct comprehensive experiments on two popular open-domain dialogue datasets and the results verify the advantages of incorporating the simulated dialogue futures. 963 2 Related Work Dialogue System. Open-domain response generation has long been the research hot spot. Recently, various efforts have been made to generate informative and diverse responses by introducing effective architectures and learning objectives and by incorporating external knowledge. Zhao et al. (2017); Gu et al. (2018) applied CVAE to model the variability of responses. Li et al. (2016b); Zhang et al. (2018a); Saleh et al. (2020) adopted reinforcement learning to encourage the model to generate desired responses through carefully designed reward functions. Zhang et al. (2018b); Chan et al. (2019); Zheng et al. (2020); Li et al. (2021) exploited persona information to improve the coherence of the response. Zhou et al. (2018); Song et al. (2019); Shen and Feng (2020) considered emotions when generating the response. Dinan et al. (2018); Lian et al. (2019); Zhao et al. (2020a,b); Li et al. (2020) conditioned the response generation model with knowledge. Different from the above works that aimed to design specific history-to-response generation models, we propose a response generation framework where possible dialogue futures are utilized in the inference phase with the help of an effective future simulation strategy. Future Modeling. There are various scenarios where considering future information is useful. In text generation, Serdyuk et al. (2017) proposed a twin network to regularize the hidden states of the left-to-right decoder with the future-aware right-to-left decoder. Zhang et al. (2018c) used the target-side hidden states generated by the right-to-left decoder to help the right-to-left decoder during translation so that the target-side future information can help avoid under-translation. Different from these works that consider the right side tokens as the future for the current token, we define future" as the next dialogue utterance of the current response in a dialogue session. In response generation, Feng et al. (2020a) proposed to use gold futures as the conditions of two discriminators and adopted adversarial training to encourage diversity. Feng et al. (2020b) employed gold dialogue futures to learn a future-aware teacher model and transferred the knowledge to a history-to-response student model via imitation learning. These works only use the future information in the training phase, while we utilize the simulated dialogue future in the inference phase to provide the history-to-response generation model with direct help. DialoGPT (Zhang et al., 2020) is a GPT-based response generation model pre-trained on large-scale open-domain dialogue corpus by maximizing the likelihood of the successive dialogue utterances (i.e., the forward dialogue flow) given the initial dialogue history. While trained on the same corpus with the same architecture, DialoGPT-MMI is trained on the backward dialogue flow where the order of the utterances in a dialogue are reversed. We adopt DialoGPT as the forward generator and DialoGPT-MMI as the backward generator. GRADE (Huang et al., 2020) is a graph-enhanced dialogue evaluation model that uses both utterance-level contextualized representations and topic-level graph representations to evaluate the response. As it is one of the SOTA dialogue evaluation models, we choose it as our dialogue selector. Our framework consists of a forward generator GF that models the history-response-future dialogue flow, a backward generator GB that models the reversed dialogue flow, a dialogue selector S that ranks the sampled utterances conditioned on the dialogue context, and a gate g that dynamically balance the ensemble weights between GF and GB . Given a history h , we first use GF to sequentially sample the response r and the future f with the help of S . We then employ the ensemble of GF and GB using g to generate the response based on both h and f . The firstly generated responses together with the future-aware second-pass responses are finally re-ranked by S to produce the final response. Figure 2 illustrates our proposed framework. Given a dialogue history h , we first use GF to generate n responses { r i } ni =1 using top-k sampling (Fan et al., 2018). We denote these responses as the first-pass responses . Then the selector S calculates the quality scores s r  R n for all history-response pairs. The quality scores naturally form a propability distribution p r  R n over the sampled responses by using a softmax operation. We here consider the response selection procedure as sampling from such a distribution. Considering 964   ! that the responses in open-domain dialogue are often diverse and hard to evaluate by any automatic evaluation metric, only using the response with the highest quality score or probability to further simulate the future is suboptimal. Meanwhile, generating futures conditioned on all the n sampled responses is too time-consuming. Therefore, borrowing the idea from beam search (Sutskever et al., 2014) where the k -best sentence prefixes are maintained during decoding to balance the searching performance and speed, we propose to keep the k -best responses at hand while discarding the others. For each of the selected response r i , we concatenate it with h and use GF to again sample n dialogue futures { f jr i } nj =1 , where f jr i denotes the j -th future simulated from h and r i . Up to now, we obtain k  n history-response-future dialogue triplets for the same dialogue history h by simulation. We again resort to the selector S to calculate the quality scores of all the generated futures conditioned on h and their corresponding ancestral responses as { s f r 1 , . . . , s f ri , . . . , s f rk } . We consider all the generated futures in the same sampling space (i.e., the future space of the given history) and directly perform softmax over the k  n quality scores to get the future distribution. Considering that the responses used to generate the futures are not equal in quality, we additionally multiply each probability of the simulated future f jr i with the probability of its ancestral response p r i to get the final ranking scores based on which we select k -best dialogue futures. Now with the history h and k plausible dialogue futures at hand, we pursue to generate the second-pass response conditioned on both the history and the future information. Given that the simulated futures contain noise derived from error accumulation in the simulation phase, it is necessary to balance the weights between the history-conditioned GF and the future-conditioned GB when they collaboratively generate the response. Hereby we introduce a trainable gate g which takes the last hidden states from GF and GB as inputs and calculates an ensemble weighting score w using an MLP with sigmoid activation. We then generate the response  r using the per-step weighted ensemble of GF and GB conditioned on h and f : P ( r t | h, f,  r <t ;  F ,  B ,  g ) = w  P ( r t | h,  r <t ;  F ) + (1  w )  P ( r t | f,  r <t ;  B ) , (1) where the subscript t denotes the t -th token in  r and  F ,  B and  g denote the parameters of GF , GB and g respectively. Specifically, we sample n responses for the ensemble generation of h and each of the k futures, resulting in k  n future-aware responses. We denote these responses as the second-pass responses . To make full use of the k -best first-pass responses, we finally re-rank the k + k  n responses with S and consider the top-ranked response as our system outputs. Recall that there are several components GF , GB , S , and g in our framework. Although some of them can directly be used without post-training, this might be suboptimal. For one thing, post-training the models on domain-specific data with the same objective often brings better performance (Guru-rangan et al., 2020). For another, the original loss functions may not be thoroughly in accord with the ultimate goal in our framework. Thereby we propose a customized joint training algorithm. For GF and GB , we adopt a similar training objective used by Zhang et al. (2020). Take GF for example, we consider every consecutive three utterances in a dialogue session as a history-response-future triplet and fine-tune the models by minimizing the negative log likelihood of the response and the future conditioned on the history. GB is fine-tuned in a similar manner with the reversed inputs. After fine-tuning, GF and GB are fixed. For g , we can directly minimize the negative log-likelihood of the gold response r  : L 1 (  g ) =  (cid:88) t log P ( r  t | h, f, r  <t ;  F ,  B ,  g ) , (2) where f is simulated from either the gold response (denoted as the teacher-forcing mode) or a sampled response (denoted as the free-running mode). While for S , considering that the original objective used in Huang et al. (2020) is not customized for selecting better responses and futures, it is better to perform task-specific post-training. Therefore, we propose to directly optimize S to our ultimate goal which is to maximize the log-likelihood of the gold response given the history and the selected simulated future. Since the sampling operation is non-differentiable, we use REINFORCE (Williams, 1992) with a self-critic (Rennie et al., 2017) baseline to estimate the gradient. We consider the future simulation process as sequential sampling from the score distributions of the responses and the futures respectively. Given the n responses generated by GF conditioned on h , we sample a response r i from p r . Then we generate n futures conditioned on h and r i using GF and again sample a future f jr i from them. We feed this sampled future and the gold history into our ensemble generation model and calculate the log-likelihood of the gold response, which is the opposite number of Equation 2, as the reward R . To reduce the variance of gradient estimation, we introduce a self-critic baseline. Concretely, we sequentially select the response and the future with the highest scores in each sampling step and calculate the reward of using the greedy future as the baseline reward R b . The gradients are then estimated as follows:   SL 2 (  S )   ( R  R b )   S [log P ( r i | h ;  S ) + log P ( f jr i | r i , h ;  S )] . the futures generated from the sampled responses may contain much noise. A better choice is to allow the model to gradually learn from easy to hard. We create a curriculum schedule (Bengio et al., 2015) that gradually switches from the teacher-forcing mode to the free-running mode. Specifically, let  denote the proportion of teacher-forcing mode, we gradually decrease  from  to  with cosine annealing schedule, where 0   <   1 . For the overall training, we first train g using Equation 2 and set  =  . Then we tune S using Equation 3 with the help of the above curriculum learning schedule. Finally, we jointly tune g and S with a fixed  =  . To verify the effectiveness of our proposed response generation framework, we experiment on two popular dialogue datasets, DailyDialog (Li et al., 2017) and PersonaChat (Zhang et al., 2018b). We follow the original train/dev/test di-vision and reconstruct the datasets by treating each consecutive three utterances as a triplet that represents history-response-future, resulting in approximately 65k/6k/6k examples in DailyDialog and 114k/14k/13k examples in PersonaChat. Posterior-GAN (Feng et al., 2020a) and RegDG (Feng et al., 2020b) are two non-GPT-based response generation models that use dialogue futures in the training time through either adversarial training or knowledge distillation. DialoGPT F denotes the fine-tuned DialoGPT medium (Zhang et al., 2020) on two downstream datasets. DialoGPT F,rerank is its enhanced version which is equipped with the dialogue evaluation model (i.e., GRADE (Huang et al., 2020)) to select the top-ranked response. ProphetChat k =? denotes the model with the same model parameters but different beam sizes when simulating the futures. ProphetChat first and ProphetChat second are used to denote the settings where only the first-pass or the second-pass responses are used in the final re-ranking process. ProphetChat w/o history means we utilize the top-ranked simulated future to generate the response without the help of the history. ProphetChat w/o selector denotes the model where we sequentially sample the responses and the futures randomly without using the selector. ProphetChat w/o train means we directly utilize the fine-tuned GF , GB and the fixed S without post-training. We manually choose a fixed ensemble weight for the ensemble generation process instead of using a trainable gate. ProphetChat w/ gold future denotes the model that utilizes the history and the gold future, which is inaccessible in the inference phase, to generate the response. Our implementation is based on the open-source toolkit Transformers (Wolf et al., 2020). For the generator GF and GB , we initialize them with the publicly released DialoGPT medium and DialoGPT-MMI medium 1 . For the dialogue selector, we use the pre-trained GRADE 2 as initialization. We firstly use AdamW (Loshchilov and Hutter, 2017) with learning rate 3e-5 to fine-tune GF and GB . Then, we jointly train the ensemble gate g and the top non-transformer layers of the selector S with learning rate 2e-5, while keeping other parameters (i.e., GF , GB and most of the parameters of S except for its top layers) fixed. We set the curriculum hyperparameters (  ,  ) as (0.0, 1.0) on both datasets. We fix the sample number n of both the response and the future as 10 and vary the simulation beam size k  { 1 , 2 , 3 , 5 } . We choose k = 5 on DailyDialog and k = 3 on PersonaChat. We use top-k sampling (Fan et al., 2018) to generate the first-pass responses, the futures, and the second-pass responses with the temperature as 0.7 and k as 40. All the hyperparameters are chosen depending on 1 https://github.com/microsoft/DialoGPT 2 https://github.com/li3cmz/GRADE 967 Models Readability kappa Sensibleness kappa Specificity kappa Posterior-GAN 0.58 0.42 0.46 0.49 0.21 0.58 RegDG 0.60 0.45 0.51 0.59 0.27 0.50 DialoGPT F 0.68 0.52 0.64 0.61 0.44 0.52 DialoGPT F,rerank 0.69 0.50 0.69 0.60 0.45 0.64 ProphetChat 0.71 0.52 0.75 0.53 0.49 0.49 Models Readability kappa Sensibleness kappa Specificity kappa Posterior-GAN 0.64 0.58 0.50 0.65 0.24 0.48 RegDG 0.65 0.56 0.53 0.55 0.28 0.63 DialoGPT F 0.69 0.59 0.68 0.66 0.42 0.52 DialoGPT F,rerank 0.70 0.44 0.72 0.52 0.48 0.52 ProphetChat 0.72 0.43 0.77 0.61 0.53 0.54 Table 2: Human evaluation results on DailyDialog (the upper) and PersonaChat (the lower) datasets. Automatic Metrics. We use BLEU (Papineni et al., 2002) to measure the word overlap between the ground truth responses and the generated ones. For simplification, we use B-n to denote the n-gram overlap scores. We employ Distinct 1-4 (Li et al., 2016a) to measure the diversity of the generated responses, where Distinct-n (abbreviated as D-n ) represents the ratio of distinct n-grams in responses. We adopt the embedding-based metrics (i.e., Average , Extrema , and Greedy ) (Liu et al., 2016) to measure the semantic relevance between the ground truth responses and the generated ones. Human Evaluation. We ask three well-educated annotators to score 150 randomly selected responses generated by ProphetChat and other baselines. The annotators are asked to evaluate the human-likeness of the responses from three perspectives: readability, sensibleness and specificity. For readability, we ask annotators whether the response is grammatically correct and easy to read. For sensibleness and specificity, we follow Adiwar-dana et al. (2020) to conduct the evaluation. For all three metrics, the annotators are asked to give 0-1 labels. We provide the averaged scores and further calculate the Fleiss's kappa (Fleiss, 1971) to measure the inter-annotator agreement. Table 1 presents the overall performance of our proposed method as well as its variants and ablations. Table 2 shows the human evaluation results. Compared with the two non-GPT baselines, GPT-based models generally achieve superior performance, especially in Distinct and human evaluation. ProphetChat outperforms all the baseline methods by a large margin on both datasets in almost all automatic metrics and all human evaluation metrics. For human evaluation results, the Fleiss's kappa scores are mainly distributed in [0.4, 0.6], which 968 means annotators achieved moderate agreement. With the same model parameters, we have several model variants by using different hyperparameters or computation flow in the inference phase. Here we mainly discuss two types of model variants: (1) the model with different simulation beam size k , (2) the final re-ranking among the first-pass responses or the second-pass responses. The simulation beam size. When simulating the dialogue future, we can choose different beam sizes to balance the computation cost and the performance. We test k  { 1 , 2 , 3 , 5 } on both datasets and find k = 5 is better than others on DailyDialog, while k = 3 is enough on PersonaChat. It can be seen that when k is small, increasing k can boost the performance. With the appropriate choice of the simulation beam size, ProphetChat can be deployed to various scenarios with different computation resources. We further directly test the future simulation performance by comparing the futures generated by our method and generated using the gold responses. The results are listed in Table 3. It can be observed that on DailyDialog, with the increase of k , our future simulation method gradually catches up with the teacher forcing counterpart in BLEU and Distinct, while still lagging behind in embedding-based metrics. On PersonaChat, our method even outperforms TF rerank in several metrics. These findings proves that we are able to obtain dialogue futures of good quality solely based on the history through our effectiveness future simulation algorithm. Re-ranking among the first pass or the second pass responses. Recall that in our main framework, we finally gather the k first-pass responses and the k  n second-pass responses together and finally re-rank them with the selector conditioned on both the history and the corresponding future. From Table 1 we can find that on both datasets, re-ranking using both groups of responses yield better performance in most of the metrics than only using one of them. When comparing their individual performance, it can be observed that on DailyDialog, ProphetChat second is superior to ProphetChat first in BLEU and Distinct, while ProphetChat first wins in embedding-based metrics. On PersonaChat, ProphetChat second wins all metrics except BLEU. There exist some cases where the simulated futures are meaningless or include irrelevant information. When this happens, the final re-ranking process comes as the remedy. We find that the proportions of the test cases where the final responses are picked from the second-pass responses are 40.4% on DailyDialog and 36.6% on PersonaChat, which are less than the proportions of the second-pass responses involved in re-ranking. This finding indicates that re-ranking plays a vital role to select the appropriate responses from the two groups of candidate responses of various qualities. We make ablation study from several perspectives including the effect of the history, the selector and the training algorithm. Table 1 shows that although only using the simulated future (i.e., ProphetChat w/o history) can generate plausible responses, the performance is largely inferior to the full model. Also, we observe that ProphetChat w/o selector underperforms the full model, demonstrating the effectiveness of the selector which helps simulate better futures. When considering the training objective, we find that ProphetChat w/o train already achieves good performance, but jointly training the whole model further makes our model perform better. Finally, when provided with the gold future, ProphetChat w/ gold future outperforms ProphetChat in terms of embedding-based metrics on both datasets, and BLEU on PersonaChat, while underperforming on other metrics. In other words, with the simulated futures, ProphetChat can achieve comparable performance with the model that "cheats" to see the gold future, which also demonstrates the effectiveness of our method. Figure 3 presents two cases sampled from the two datasets. For ProphetChat, in addition to its final response, we provide its corresponding first-pass response, and the simulated future of the response. From the two cases we can observe that by taking the simulated future into consideration, ProphetChat generates more informative responses than baselines. Specifically, in case 1, when a two-choice query is issued in the history, the first-pass response chooses online" as the answer and meanwhile poses another question. ProphetChat then uses the history and this first-pass response to simulate the future where the other possible choice (i.e., the bookstore) is talked about. Given the history and the simulated future, ProphetChat finally obtains its response which not only answers the query 969 Case 1 History: That is cool. in the history but also incorporates the cues in the future. This response becomes more informative than the previous one. A similar phenomenon can also be found in case 2 where the final response is more comprehensive that connects the history and the future smoothly. We propose a novel response generation framework that utilizes the simulated dialogue futures in the inference phase to enhance response generation. To acquire the dialogue futures, we design an effective beam-search-like roll-out strategy using a history-to-response dialogue generation model and a dialogue selector. To make use of the simulated future, we use the dynamic ensemble of the history-to-response and the future-to-response generation model. Experiment results demonstrate the effectiveness of our proposed method on two popular datasets. In the future, we plan to enable our future simulation method to simulate multiple turns of dialogue futures. We would like to thank the anonymous reviewers for their constructive comments. This work was supported by the National Key Research and Development Program of China (No. 2020AAA0106600), National Natural Science Foundation of China (NSFC Grant No. 62122089 and No. 61876196), and Beijing Outstanding Young Scientist Program (No. BJJWZYJH012019100020098). This paper proposes a new dialogue generation framework that utilizes the simulated dialogue futures in the inference phase to enhance the generation of the response. Generative approaches are widely used in a wide range of dialogue applications. The proposed method improves the quality of the generated responses, which could be beneficial to research and real-world applications. The research will not pose ethical issues. This paper doesn't involve any data collection and release thus there are no privacy issues. All the datasets used in this paper are publicly available and are widely adopted by researchers to test the performance of open-domain response generation models. This paper conducts human evaluation to evaluate the quality of the generated responses. Three part-time research assistants were recruited to do human evaluation with clearly demonstrated evaluation rules. They worked with the pay 100 CNY/hour during their evaluation.
abstain abstain abstain objective method method abstain abstain abstain abstain abstain result abstain abstain abstain abstain other abstain abstain method abstain abstain method abstain method method method method abstain method method method abstain abstain method abstain abstain abstain method method method method method method method abstain abstain method method method method method method abstain abstain method method method method abstain abstain method abstain method method method method method method method method abstain abstain abstain method method method method method method method abstain abstain abstain abstain abstain abstain method method method method abstain method method method method method method method method method other abstain method method method method abstain method abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result result result abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain objective method method objective method other other abstain abstain abstain abstain abstain method method abstain abstain