Responsing with image has been recognized as an important capability for an intelligent conversational agent. Yet existing works only focus on exploring the multimodal dialogue models which depend on retrieval-based methods, but neglecting generation methods. To fill in the gaps, we first present a new task: multimodal dialogue response generation (MDRG) given the dialogue context, one model needs to generate a text or an image as response. Learning such a MDRG model often requires multimodal dialogues containing both texts and images which are difficult to obtain. Motivated by the challenge in practice, we consider MDRG under a natural assumption that only limited training examples are available. Under such a low-resource setting, we devise a novel conversational agent, Divter, in order to isolate parameters that depend on multimodal dialogues from the entire generation model. By this means, the major part of the model can be learned from a large number of text-only dialogues and text-image pairs respectively, then the whole parameters can be well fitted using just a few training examples. Extensive experiments demonstrate our method achieves state-of-the-art results in both automatic and human evaluation, and can generate informative text and high-resolution image responses. With the development of instant messaging technology in the recent decades, the intermediary of online conversation has also changed from pure text to a variety of visual modalities (e.g., image, gif animation, short video). Similar to communicating by the messenger tools (e.g., Facebook, WhatsApp, WeChat) in reality, an excellent intelligent conversational agent should not only be able to converse freely with plain text, but also have the ability to perceive and share the real visual physical world. Although recently some large-scale pre-trained text-only dialogue generation models, such as DialoGPT (Zhang et al., 2020), Blender (Roller et al., 2021), Meena (Adiwardana et al., 2020), have shown excellent performance, they still cannot rely exclusively on plain text to completely simulate the rich experience of visual perception. Recently, various vision-language tasks have been introduced and attracted widespread attention, such as visual question answering (Ren et al., 2015; Lu et al., 2016; Anderson et al., 2018; Li et al., 2019a; Huang et al., 2020), image captioning (Xu et al., 2015; Anderson et al., 2016; Ghanimifard and Dobnik, 2019; Cornia et al., 2020), image-grounded dialogue (Das et al., 2017; Yang et al., 2021; Agarwal et al., 2020; Qi et al., 2020; Chen et al., 2021; Liang et al., 2021). Specifically, in human conversations, the images can easily show rich visual perception, which is hard to be expressed by plain text. As the example shown in Figure 1, images are required in at least three circumstances: (i) the other speaker has little knowledge (e.g., colorful Burano, in the 1st image) of the objects only you had seen; (ii) to share more details (e.g., red wine and pasta, in the 2nd image) of the objects even you have common knowledge of them; (iii) to express your emotions (e.g., happy, in the 3rd image) about a specific event. An existing related task is photo sharing (Zang et al., 2021), which aims to select and share the image based on the textual context, is a challenging task that requires models to understand the background story which complemented by human imaginations, rather than to locate related visual objects or explicitly mention main visible content in the image as the previous works do. Zang et al. (2021) propose a retrieval-based method to resolve the above challenge. However, the performance of the retrieval-based method is limited in specific domains by the size of the pre-constructed conversational history repository, especially for long-tail contexts that are not covered in the history, where the set of image responses of a retrieval system is also fixed. On the other hand, a better way is to generate a new one accordingly. In this paper, we formulate a new problem: M ultimodal D ialogue R esponse G eneration ( MDRG ), that is, given the dialogue context, the model should not only generate a pure text response but also have the capacity to generate a multimodal response (e.g., containing both image and text). We argue that there are still some hindrances to application, since (1) the sophisticated neural end-to-end architecture will overfit to very few well-annotated training data (e.g., a few existing 10k multimodal dialogues). Evidence is that when discussing the topics outside the training data domain, its performance drops dramatically; and (2) as human effort is expensive, it is not easy to collect enough training data for a new domain. Based on the above facts, we take a step further to extend the assumption of MDRG to a low-resource setting where only a few multimodal dialogues are available. To tackle the above challenges, our key idea is to make parameters that rely on multimodal dialogues small and independent by disentangling textual response generation and image response generation, and thus we can learn the major part of the generation model from text-only dialogues and <image description, image> pairs that are much easier to be obtained. Specifically, we present Divter , a novel conversational agent powered by large-scale visual world experiences. As shown in Figure 2, our Divter is made up of two Transformer-based (Vaswani et al., 2017a) components: a multimodal dialogue response generator, and a text-to-image translator. Divter takes the dialogue context as input, then generates a textual sequence which may contains a text response or a textual image description or both of them. The text-to-image translator takes above image description as condition, then generates a realistic and consistent high resolution image. Both components are independent with the opposite knowledge, and thus can be pre-trained using a large number of text-only dialogues and 2855 the <image description, image> pairs respectively. The end-to-end Divter depends on the multimodal dialogues constructed as the tuple: ( dialogue context, text response / <image description, image> ) , but the joint learning and estimation of the two components just require a few training examples depending on specific domains. Contributions of this work are three-fold:  To the best of our knowledge, it is the first work on the multimodal dialogue response generation. We explore the task under a low-resource setting where only a few multimodal dialogues are assumed available. We present Divter, a novel conversational agent which can effectively understand dialogue context and generate informative text and high-resolution image responses. Extensive experiments on PhotoChat Corpus (Zang et al., 2021) indicate the effectiveness of Divter, it achieves a significant improvement with pure text dialogue generation model and retrieval-based image sharing method. End-to-end response generation for textual open-domain dialogues is inspired by the successful application of neural sequence-to-sequence models on machine translation (Sutskever et al., 2014). On top of the basic architecture (Shang et al., 2015; Vinyals and Le, 2015), the vanilla encoder-decoder method is widely extended to address the critical challenges in open-domain dialogue systems, including improving the diversity of responses (Li et al., 2016a; Zhao et al., 2017; Tao et al., 2018), modeling conversation contexts (Ser-ban et al., 2016; Xing et al., 2017; Zhang et al., 2019; Zhao et al., 2020), controlling attributes of responses (See et al., 2019; Zhou et al., 2018; Xu et al., 2019), biasing responses to some specific personas (Li et al., 2016b; Zhang et al., 2018), incorporating extra knowledge into generation (Di-nan et al., 2019; Ghazvininejad et al., 2018; Kim et al., 2020; Li et al., 2020), and building general pre-trained agents (Adiwardana et al., 2020; Zhang et al., 2020; Roller et al., 2021; Qi et al., 2021). Different from the previous works on open-domain dialogue response generation that converse freely with plain text, our work lies in the research of multimodal response generation. In the research of text-to-image generation, various works have been extensively studied. Mansimov et al. (2016) shown the Draw generative model (Gregor et al., 2015) could generate images from natural language descriptions. Reed et al. (2016) proposed a generative adversarial network to improve the image fidelity. Then some improvement methods continue to optimize the generation architecture, such as stacked generators (Zhang et al., 2017), attentional network (Xu et al., 2018), and extra knowledge (Li et al., 2019b). Nguyen et al. (2017) provided a unified probabilistic interpretation of related activation maximization methods to produce high-quality images at higher resolutions. Separately, Cho et al. (2020) used uniform masking with a large range of masking ratios and align the suitable pre-training datasets to the proper objectives. More recently, Ramesh et al. (2021) and (Ding et al., 2021) adopt transformer-based methods which autoregressively model the text and image tokens as a single stream of data. For this multimodal response generation scenario, we use the textual image description to bridge above textual dialogue generation and text-to-image generation models, where the image description is the output of the former and input of the latter in a low-resource setting. Suppose that we have dataset DS = {( U i , R i )} ni = 1 , where  i  { 1 , . . . , n } , U i = { u i, 1 , . . . , u i,n i } is the dialogue context with u i,j the j -th utterance, and R i is the response regarding to U i . u i,j and R i could contain two modalities: text, and image. The goal is to learn a generation model P ( R  U ;  ) (  denotes the parameters of the model) with DS . Thus, given a new dialogue context U , one can generate a multimodal response R following P ( R  U ;  ) . This section first formulates the unified tokenization method for multimodal dialogues. We then introduce the two important components in our proposed multimodal dialogue response generation model ( Divter ) under low-resource scenario, including (i) textual dialogue response generator; (ii) text-to-image translator. Figure 2 shows the overall of our Divter . To learn a multimodal generation model, we should first model the unified representations of both text and image. Inspired by the success of DALLE (Esser et al., 2020) and VQGAN (Ramesh et al., 2021), to utilize the highly expressive transformer architecture for text-to-image generation, we need to express an image in the form of a sequence, similar to what we usually do for pure text tokenization. The tokenization for text is already well-studied, e.g., BPE (Gage, 1994). This work uses 50257 BPE-encoded tokens and distributed embedding of Transformer architecture (Vaswani et al., 2017b) to model the texts in a dialogue. The tokenizer for image is a discrete Auto-Encoder (VQGAN 1 ) V as shown in Figure 2. V uses an encoder VE to compress each image r v of shape H  W  3 into  z of shape h  w  d z , then each vector of dimension d z would be quantized to its closest embedding z k in a learned, discrete codebook Z = { z k } Kk = 1  R d z under the action of element-wise quantization q (  ) z q = q (  z ) = ( arg min z k Z   z ij  z k )  R h  w  d z (1) Thus r v can be represented by a spatial collection of codebook entries z q  R h  w  d z . The decoder VD maps the z q back to a image  r v to reconstruct the input. In this work, H = W = 256 , h = w = 16 , K = 16384 , d z = 256 . The learning details of V and Z could be found in Ramesh et al. (2021). Learning an effective multimodal generation model with a single sequence-to-sequence model often requires a large number of training instances. However, only very few multimodal dialogues are available due to the privacy restrictions on social media and the expensive human effort. On the other hand, as shown in Figure 3, there existed a large number of open source text-only dialogues (e.g. Reddit comments 2 , formulated as DC = {( U i , r ei )} Ni = 1 with ( U i , r ei ) a <text dialogue context, text response> pair) , and a large number of <image description, image> pairs (e.g. YFCC100M (Thomee 1 https://github.com/CompVis/ taming-transformers 2 https://files.pushshift.io/reddit/ Figure 3: Abstract Logic of the proposed approach. Solid lines mean that there exists large-scale training set to pre-train the generation model, while dotted lines mean that only very few training instances are available,    means bad generation quality. et al., 2016), formulated as DP = {( c j , r vj )} Mj = 1 with ( c j , r vj ) a <textual image-description, image> pair). Based on the above facts and the low-resource challenges on MDRG task, we adapt to incorporate generative text-to-image translation into text-only open domain dialogue response generation. More specifically: (i) if the multimodal dialogue context contains an image, we replace the image with its description to form a text-only context, and take this context as the input of the text-only dialogue generation model G (pre-trained with DC ); (ii) if we need to generate an image as a part of response, we could first generation a textual description with G , then adopt a text-to-image translator module F (pre-trained with DP ) to translate the description to a synonymous image. To bridge G and F , we further extend the formalization of DS to a new D S in which each image r v is paired with its textual description c . Both the (i) and (ii) actions can be independently learned, which becomes the key to aiding the small D S with the large DC and DP . By this means, the current goal is to learn a generation model P ( R  U ;  ) with D = { D S , DC , DP } . With the pre-trained G and F available, we finally use D S to jointly finetune G and F to obtain the capacity of generating multimodal responses. Figure 2 illustrates the architecture of our model. The model is made up of two components: a textual dialogue response generator G and a text-to-image translator F . In the rest of this section, we will elaborate these two modules in detail. The textual dialogue response generator G is a sequence-to-sequence model based on the Trans-2857 former architecture (Vaswani et al., 2017b), it consists of a 24-layers Transformer with a hidden size of 1024 and 16 heads. Specifically, given a text dialogue context U = { u 1 , . . . , u l } from  DS as source, and the target is a text  R = { w 1 ,  , [ SEP ] , [ DST ] ,  , [ SEP ] ,  , w T } with w t the t -th word, the [DST] token means the following subsequence is a textual image description c . The generation loss is defined by LG = E ( U,  R ) DS [ log p (  R )] (2) p (  R ) =  t p ( w t  U, w 1  t  1 ) (3) Inference Given a new text dialogue context U , when a generated image description c occurs, it will be fed into the following text-to-image translator, then constructed to the codebook embeddings of its synonymous image. The text-to-image translator F is also a sequence-to-sequence generation model based on the Transformer architecture, it consists of 24-layers Transformer with a hidden size of 1024 and 16 attention heads. Given an image r v  RH  W  3 and its textual description c = { w 1 ,  , w T } from  DS , with the VE and Z available, we can represent r v in terms of the codebook indices of its encodings. More precisely, the quantized encoding of image r v is given by z q = q (VE ( r v ))  R h  w  d z , and could be transferred to a sequence s  { 0 ,  , Z  1 } h  w of indices from the codebook Z , which is obtained by replacing each code with its index in the codebook Z s i,j = k such that ( z q ) i,j = z k (4) Then we concatenate tokenized c and s to a single stream of tokens x = { w 1 ,  , w T , [ SEP ] , s 1 ,  , s h  w } (5) and train an autoregressive transformer to model the joint distribution over the text and image tokens, the generation loss is defined by LF = E ( c,r v ) DS [ log p ( x )] (6) p ( x ) =  t p ( w t  w 1  t  1 )  i p ( s i  c, s 1  i  1 ) (7) Inference Given a description c , we leverage the text-to-image translator to generate the representations  z = F( c )  R h  w  d z of its synonymous image. Let us denote {  g ,   ,   } as the parameters of textual dialogue response generator G , image tokenizer V and text-to-image translator F . In the pre-training stage, we use textual dialogues DC to estimate  g , use the ImageNet (Deng et al., 2009) to estimate   , use <image description, image> pairs DP to estimate   . Then we fix   , and jointly finetune  g and   with D S , thus the final objective is to minimize the integrated loss L = LG +  LF (8) where  is a hyper parameter. Remarks. In this work, we mainly focus on integrating text and image responses generation, but our proposed approach actually provides a recipe for a general solution to low-resource MDRG in which the target modality could be gifs, videos, or speech sounds, etc. To do that, one only needs to modify the text-to-image translator to make it compatible with the specific modality type, then pre-train a new text-to-<target modality> translator. To evaluate the performance of Divter, we conduct comprehensive experiments on the PhotoChat dataset released by Zang et al. (2021), which is a multimodal conversational dataset consisting of 10917 images and 12286 dialogues, each of which is paired with a user image that is shared during the conversation, and each image is paired with its text description. The dataset has been split into 10286 train, 1000 dev, and 1000 test instances. More details are described in Appendix A.1. We conduct evaluation with both automatic metrics and human judgements. For automatic evaluation, we focus on four aspects: (1) Image Intent Prediction, the goal of this task is to predict whether a image should be produced in the next turn for given context; (2) Text Description Generation; (3) Image Generation Quality ; (4) Text Response Generation. For (1), we follow Zang et al. (2021), which formulates the problem as a binary classification task, and use F1 as metric; for (2) and (4), we use PPL , BLEU (Papineni et al., 2002), Rouge (Lin, 2004) and F1 ; for (3) we follow Ramesh et al. (2021) and use Frechet Inception Distance ( FID ) and Inception Score ( IS ). For human evaluation, we randomly sample 200 dialogue contexts and generate responses from PhotoChat for Divter and baselines. Three human annotators are asked to score the response quality on a scale of {0, 1, 2} from four aspects: (1) Context Coherence : Whether the text response is coherent with the context; (2) Text Fluency : Whether the text response is natural and fluent; (3) Image Quality : The quality (including definition and integrity) of the image response; (4) Background Consistency of Image : For each dialogue, We select the top-8 generated/retrieved images group and ask the annotators to decide whether the group is consistent with the dialogue background, a qualitative assessment is also shown in Figure 5. We report the average scores over three annotators, and the higher score means the better. We also compare both pure text Divter and multimodal Divter with DialoGPT, respectively. The pure text Divter means we block the [DST] token in the vocabulary in the decoding stage, so that the responses would only contain texts. We also randomly sample 200 dialogues. To each annotator, two responses from different models are presented, which are randomly shuffled to hide their sources. The annotators then judge which response is more effective in improving the dialogue experience and attractiveness. The agreement among the annotators is measured by Fleiss' Kappa (Fleiss, 1971). For the textual dialogue response generator G , we use DialoGPT (Zhang et al., 2020) as pre-trained model initialization, trained on 147M conversation-like exchanges extracted from Reddit comment chains over a period spanning from 2005 through 2017. In the fine-tuning stage, we concatenate the context turns with the token [SEP] as a single sequence, we adopt Adam optimizer as an initial learning rate of 1e-5, and the batch size is 256, the training of PhotoChat is conducted on 16 Nvidia Tesla V100 32G GPU cards. We use beam search(size=5) to decode the text sequence. For the text-to-image translator F , we randomly select 5M <categorical image description, image> pairs from ImageNet, and <image description, image> pairs from YFCC100M (Thomee et al., 2016) as training data. We set the maximum image description length as 32, then pre-train F for 3.5 million steps with a batch size of 256 accumulated on 16 Nvidia Tesla V100 32G GPUs. In the fine-tuning stage, we train PhotoChat for 50000 steps. In the joint learning, we first train F for 48000 steps, then jointly train G and F for 2000 steps. The  in Eq.8 is 0.2. Early stopping on validation is adopted as a regularization strategy. All the hyper parameters are determined by grid search. More details are described in Appendix A.3. We implement the image Auto-Encoder using the code https://github.com/CompVis/ taming-transformers , implement the Textual Dialogue Response Generator using the code https://github.com/microsoft/ DialoGPT , and implement the Text-to-Image Translator using the code https://github. com/lucidrains/DALLE-pytorch . Two pre-trained models BERT-base (Devlin et al., 2019) and T5-3B (Raffel et al., 2020) are selected as baselines to measure the Image Intent Predic-tion task in Section 5.2. They takes the text dialogue context as input, and predict whether a image will be shared in the next turn. SCAN is proposed by Lee et al. (2018), the model captures interplay between image regions and text tokens to infer image-text similarity, SCAN achieves state-of-the-art performance of the Image Retrieval task on PhotoChat. S2S-TF is a single sequence-to-sequence model with 24-layers Transformer, we only use PhotoChat to train this multimodal generation model. As shown in Table 1, our Divter achieves not only comparable performance with the state-of-the-art retrieval-based image response intent prediction model but also achieves remarkable performance in all the generation metrics. This indicates that Divter can accurately judge the timing of generating image response with the given dialogue context, and produce text responses that are coherent to the context, and generate high-quality image responses. The significant performance gap between Divter and the baseline models (e.g. S2S-TF, Divter variants) without pre-training indicates the superiority of our proposed learning strategy. Table 2 reports the results of human evaluation, our Divter also significantly outperforms the baselines on most of the aspects. The comparison results shown in Table 3 indicates (1): out Divter can achieve comparable performance on pure text response generation with DialoGPT; (2): the multimodal responses generated by Divter achieve a significant improvement on the dialogue experience and attractiveness in contrast to pure text dialogue model (DialoGPT). We conduct extensive ablation experiments over different variants to better understand their relative importance to the MDRG task. As shown in Table 1, all the variants lead to worse performance 2860 Example 1 Example 2 A: OMG...the new ice cream shop is amazing. in most of the metrics. For a more intuitive comparison, the qualitative assessment results are also shown in Figure 4. In particular, both quantitative and qualitative results on the ablation study validate that: (1) pre-training is crucial to low-resource multimodal dialogue response generation, since removing any component from pre-training causes performance drop when training data is small; (2) in terms of impact to performance of image generation, F > G , in terms of impact to performance of text generation, G > F ; (3) The joint learning also has contributions to Divter, indicating that leveraging the integrated learning of textual context and visual image benefits more in contrast to any single one of them. To further investigate the quality of multimodal responses generated by Divter, we show two examples on the PhotoChat test data in Table 4. The given context of the first one is about ice cream, and the second one is about honey bee. As we can see, Divter can not only generate a realistic high-resolution image which is coherent to the background, but also generate the informative text responses grounded on the image. Separately, The high-quality generated images are comparable to those real-world ground truths, which demonstrates the practicability of Divter. Benefits over retrieval-based methods To further investigate and compare the generalization capability between Divter and the retrieval-based method, we also get top-10 generated images from Divter and equivalent retrieved images from SCAN model given the same context. As shown in Figure 5, on the one hand, the diversity and richness of the generated images are desirable, on the other hand, those retrieved results often suffer from wrong consistency with dialogue background. For example in the second case, the dialogue is talking about coffee, but the retrieved images contain some uncorrelated objects like milk, cake, dog' and snack. And in the third example, all the retrieval results are mistaken since there is little curtain in the training and retrieval space. This demonstrates the fact that the performance of retrieval-based method is extremely limited in specific domains by the size of the pre-constructed conversational history repository, especially in the low-resource scenario. Furthermore, our proposed generation based method shows better generalization capability to tackle the low-resource challenge. In this paper, we explore multimodal dialogue response generation under a low-resource setting. To overcome the challenges from the new task and insufficient training data, we propose Divter, a neural 2861 conversational agent which incorporates text-to-image generation into text-only dialogue response generation, in which most parameters do not rely on the training data any more and can be estimated from large scale textual open domain dialogues and <image description, image> pairs. Extensive experiments demonstrate Divter achieves state-of-the-art results in automatic and human evaluation. In the future, we will explore more efficient methods to inject more modalities into response generation. We thank anonymous reviewers for their insightful suggestions to improve this paper.
abstain abstain objective abstain objective objective abstain objective abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain objective result objective result abstain abstain abstain abstain objective objective objective objective other other abstain other other other other other other other method abstain abstain abstain abstain abstain method abstain abstain method method method abstain method abstain other abstain method abstain abstain abstain other method abstain method method method abstain abstain abstain abstain method method abstain method abstain abstain other other abstain method method method method method abstain method abstain abstain abstain abstain abstain abstain other abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain other abstain abstain result abstain abstain abstain abstain abstain abstain abstain objective objective abstain objective other