Arguably, the visual perception of conversational agents to the physical world is a key way for them to exhibit the human-like intelligence. Image-grounded conversation is thus proposed to address this challenge. Existing works focus on exploring the multimodal dialog models that ground the conversation on a given image. In this paper, we take a step further to study image-grounded conversation under a fully open-ended setting where no paired dialog and image are assumed available. Specifi-cally, we present Maria, a neural conversation agent powered by the visual world experiences which are retrieved from a large-scale image index. Maria consists of three flexible components, i.e. , text-to-image retriever, visual concept detector and visual-knowledge-grounded response generator. The retriever aims to retrieve a correlated image to the dialog from an image index, while the visual concept detector extracts rich visual knowledge from the image. Then, the response generator is grounded on the extracted visual knowledge and dialog context to generate the target response. Extensive experiments demonstrate Maria outperforms previous state-of-the-art methods on automatic metrics and human evaluation, and can generate informative responses that have some visual commonsense of the physical world. Building intelligent conversational agents that can not only converse freely with human but also have the ability to perceive the physical world, has been one of the longest standing goals of natural language processing (NLP) and artificial intelligence (AI). Although the recent large-scale conversation models trained on text-only corpora, such as Meena  Work performed during the internship at Microsoft. (Adiwardana et al., 2020), Blender (Roller et al., 2020) and DialoGPT (Zhang et al., 2020), have shown the compelling performance, they are still lack of the perception ability to our physical world. A recent study (Bisk et al., 2020) points out the successful linguistic communication relies on a shared experience of the world that makes language really meaningful. The visual perception is a rich signal for modeling a vastness of experiences in the world that cannot be documented by text alone (Harnad, 1990). On the other hand, human-human conversations involve their understandings of context, the background knowledge they had, and perhaps most importantly the experiences of the world they shared, e.g. , what they have seen before. Figure 1 shows a conversation between humans. Human-A recalls his/her past experience of playing volleyball or having BBQ on the beach when human-B talks about vacation on the beach of Hawaii. However, the association relationship between beach and volleyball (or BBQ) is hard to capture in traditional knowledge bases, such as knowledge graph. Motivated by this, we select a common word pizza and collect the top 17 words that mostly co-occur with pizza on Google Figure 2: The word co-occurrence distribution with pizza on Google knowledge graph and MS-COCO images. Knowledge Graph 1 and MS-COCO images 2 (Lin et al., 2014). As shown in Figure 2, the words co-occurring with pizza on knowledge graph tend to be the abstract concepts, while the co-occurrence relationship of object tags on images reflects some commonsense of our physical world, e.g. , pizza is usually on the dining table, people usually use knife when eating pizza. Interestingly, we found the pizza also co-occurs with cell phone and even plotted plant. This indicates when people eat pizza, they sometimes would put their cell phones aside on the table, or there might exist some plotted plants in the restaurant. Thus, empowering conversational agents to have the visual perception ability about the physical world is a key way for them to exhibit the human-like intelligence. The existing works (Mostafazadeh et al., 2017; Huber et al., 2018; Shuster et al., 2020) focus on exploring the multimodal dialog models that ground the conversation on a given image. Recently, Yang et al. (2020) propose to learn the dialog generation model with both image-grounded dialogs and textual dialogs by resorting to text-to-image synthesis techniques (Xu et al., 2018; Qiao et al., 2019) to restore a latent image for the text-only dialog. Even so, these works are still constrained by the assumption that the dialog is conducted center around a given (or synthesized) image. In this paper, we take a step further to extend the assumption of image-grounded conversation to a fully open-ended setting where no image-dialog pairs are assumed available. Specifically, we present Maria, a neural conversational agent powered by visual world experiences which are retrieved from a pre-built image index, e.g. , the 1 https://developers.google.com/ knowledge-graph/ 2 We calculate the co-occurrence distribution of object tags from the images in MS-COCO dataset. More examples could be found in Appendices. Open Images Dataset (Kuznetsova et al., 2018). Maria consists of three components: text-to-image retriever, visual concept detector, and visual-knowledge-grounded response generator. The retriever is responsible for retrieving a piece of visual world experiences, e.g. , a correlated image to the dialog from an image index. The visual concept detector utilizes the object detector from UpDown (Anderson et al., 2018) to extract the regions features ( i.e. , bboxes) and the corresponding visual concepts ( i.e. , tags) from the retrieval images. Hence, we can construct ( bboxes , tags , context , response ) 4-tuple as the training data. Finally, these constructed 4-tuples are used to train the visual-knowledge-grounded response generator, which is built on the top of a multi-layer Transformer architecture (Vaswani et al., 2017). To effectively inject the visual knowledge into the response generator, we carry out the Masked Concept Prediction and Visual Knowledge Bias besides the response generation objective. The former aims to align the semantic representations between textual words and image regions, while the latter tries to provide more visual knowledge to facilitate the dialog generation. The experimental results on Reddit Conversation Corpus (Dziri et al., 2019a) demonstrate that Maria significantly outperforms previous state-of-the-art methods, and can generate informative responses with visual commonsense of our physical world. Overall, the contributions of this paper are summarized as follows:  We explore the task of image-grounded dialog generation under a fully open-ended setting where no specific image-dialog pairs are assumed available, i.e., zero-resource image-grounded conversation. To the best of our knowledge, this is the first work to connect dialog corpus with the unpaired image data;  We present Maria, a neural conversational agent consisting of three flexible components, which can effectively capture the visual commonsense from images and accordingly generate informative and vivid responses;  Extensive experiments on the widely used Reddit Conversation Corpus are conducted to justify the effectiveness of Maria. Vision and Language In the research of vision and language, various tasks have been extensively studied, such as image captioning (Vinyals et al., 2015; Lu et al., 2017; Hu et al., 2020), visual question answering (Antol et al., 2015; Anderson et al., 2018), visual dialog (Das et al., 2017a,b). Popular benchmark datasets in this area include MS-COCO (Lin et al., 2014), VisDial (Das et al., 2017a) and Visual Genome (Krishna et al., 2017). Visual dialog is a task to answer the questions about the factual content of the image in a multi-turn manner. Differently, image-grounded conversation studies how to reply to a dialog context and a given image with proper responses in an open-ended way. Dialog Generation Encouraged by the success of the neural sequence-to-sequence architecture (Sutskever et al., 2014) on machine translation, end-to-end neural approaches on open-domain dialog generation (Vinyals and Le, 2015; Shang et al., 2015; Serban et al., 2016; Sordoni et al., 2015; Xing et al., 2017; Wu et al., 2018; Zhang et al., 2020; Xu et al., 2019; Adiwardana et al., 2020) have been widely studied in literature. Recently, there is an emerging trend towards grounding the dialog generation models on the external knowledge, such as knowledge graphs (Zhou et al., 2018), documents (Ghazvininejad et al., 2018; Dinan et al., 2019; Kim et al., 2020; Zhao et al., 2020a,b; Li et al., 2020) and images (Mostafazadeh et al., 2017; Shuster et al., 2020; Yang et al., 2020). Different from the previous work on knowledge-grounded conversation that connects dialogs with unpaired document knowledge (Li et al., 2020), our work lies in the research of image-grounded conversation where a response is generated with a dialog context and a given image. Existing works (Mostafazadeh et al., 2017; Shuster et al., 2020; Yang et al., 2020) in this direction assume there is a given (or synthesized) image for the dialog and explore the multimodal dialog models. In contrast to these works, we study the image-grounded conversation under Multi-turn Reddit Conversation Image Index (Open Images) Visual Concept Detector Visual-Commonsense-Aware Response Generation Model Training: (,) ; Inference:  Text-to-Image Retrieve Module Top-k Images (k=1 here): (  ,  ,,  ) Training Quaternion: ,,, ; Inference Triplet: (,,) Figure 3: The flowchart of our framework. a fully open-ended assumption where no paired dialog and image are assumed available, i.e. , zero-resource image-grounded conversation. Suppose we have a dialog set D = { ( C i , R i ) } ni =1 , where  i  { 1 , . . . , n } , C i refers to a dialog context and R i is a response to C i . We assume there is a set of images V = { V j } mj =1 , where  j  { 1 , . . . , m } , V j denotes an image. C  D , we assume that there is an image V that triggered by the given dialog context C and response R . Our goal is to estimate a generation model P ( R | V, C ) from D and V . Thus, given a new dialog context C associated with an image V , the model can generate a response R according to P ( R | V, C ) . To learn such a generation model P ( R | V, C ) , we need to tackle several challenges: (1) How to bridge the gap between unpaired dialog corpus and image data; (2) After obtaining the correlated images, how to extract the detailed visual features and concepts; (3) How to effectively inject the visual knowledge into response generator and enable it to generate responses that are visual-knowledge-grounded. Figure 3 illustrates the framework of our approach. We first build a large-scale image dataset and leverage a cross-modal matching model to retrieve a correlated image using the content of the dialog. Then an off-the-shelf object detector is applied to extracting the object features and visual concepts from the retrieval image. Finally, the response generator is trained to generate the target response conditioned on the context, extracted object features, and visual concepts. In this section, we develop a retrieval model that assigns each dialog with a correlated image V . Specifically, we train a text-to-image matching model from image captioning dataset and utilize it to construct the ( C, R, V ) triple data. Modeling To improve the efficiency of cross-modal retrieval model on large-scale dialog corpus and image dataset, we adopt a two-tower architecture (Lu et al., 2019) to accelerate the retrieval process where the image features can be pre-extracted offline. The model takes a sentence T and an image V as input, and predicts the relevance score s ( T, V ) between the sentence and the image. We use a text encoder and an image encoder to produce the representations of T and V , respectively. The text encoder is a pre-trained BERT-base model (Devlin et al., 2019) and we use the hidden state of special token [CLS] as the embedding of T : e t = BERT ( T ) (1) Then a Multi-Layer Perceptron (MLP) projects the sentence embedding into the cross-modal space. We follow Tan and Bansal (2020) to perform L2-normalization on the last output features, by which we can simplify the nearest neighbor search problem in the euclidean space to the Maximum Inner Product problem (Mussmann and Ermon, 2016): f t ( T ) = H t ( e t ) (cid:107) H t ( e t ) (cid:107) (2) Similarly, the image encoder is composed of a pre-trained ResNeXt backbone (Xie et al., 2017) and a MLP with L2 normalization: f v ( V ) = H v ( e v ) (cid:107) H v ( e v ) (cid:107) , e v = ResNeXt ( V ) (3) Thus, we define the relevance score s ( T, V ) as an inner product of the language feature representation f t ( T ) and image feature representation f v ( V ) : s ( T, V ) = f t ( T ) (cid:62) f v ( V ) (4) Training We train the cross-modal matching model on MS-COCO image captioning dataset (Lin et al., 2014), where each image is paired with 5 sentences describing its visual content. The model is optimized by minimizing the hinge loss so that the relevance score s ( T, V ) of the positive image-sentence pair can be larger than the negative pair s ( T, V  ) by at least a margin M : L hinge (cid:0) T, V, V  (cid:1) = l (cid:88) i =1 max { 0 , M  s ( T, V ) + s (cid:0) T, V  (cid:1)(cid:9) (5) Inference Given the trained retrieval model, we can now assign each dialog with a correlated image V . To ensure the diversity and richness of the retrieval results, we fetch 500,000 images from the large-scale Open Images dataset (Kuznetsova et al., 2018) as our image set V . The image V i  V with the maximum relevance score is paired with the given dialog ( C i , R i )  D . Note that for the dialog in the training set, we use both the context C and response R are concatenated as the query for retrieval ( i.e. , T = ( C, R ) ), which is beneficial to retrieving an image with the related visual knowledge. On the other hand, for the validation/test set of the dialog corpus, the query is only the context ( i.e. , T = C ) so as to keep consistent with the real-world setting where the response is unavailable and need to be generated at inference. Given the correlated image V i to the dialog as the visual clue, we can now extract the visual knowledge from it. One naive approach is to utilize the CNN-based models to extract the latent image features. However, this approach does not consider the fine-grained representation modeling for images, which is crucial for the dialog model to understand the local visual features in images. To address this issue, we adopt an object detection model (Ander-son et al., 2018) pre-trained on Visual Genome (Krishna et al., 2017) to extract a set of salient object features O = { o k } Kk =1 , where each object feature o k is a 2048-dimensional vector. These features represent the images at the level of objects and other salient regions, which has proven to be vital in many high-level image understanding tasks. Besides, the same detector is used to extract a set of visual concepts Q = { q m } Km =1 , where each concept q m is the high-precision textual label of the visual region, e.g. , sunset, melon, etc. In this manner, we simultaneously obtain the fine-grained image representations and the necessary visual concepts for the subsequent dialog generation. In this section, we propose a unified architecture to effectively inject a set of region features and corresponding visual concepts into the response generation model. In following parts, we describe the model design and training objectives in detail. Figure 4 shows the architecture of our response generation model, which is a multi-layer transformer network for both bidirectional vision/context ( O, Q, C ) encoding, and unidirectional response R decoding, via the flexible self-attention masks inspired by (Dong et al., 2019). For each token, the final input representation to the multi-layer transformer network is the element-wise summation of four kinds of embeddings, including token-level, turn-level, position-level, and segment-level. Then, we concatenate all the input representations to one sequence for model training. Token-Level The token-level embeddings are the concatenation of ( O w , Q w , C w , R w ) , which denote the token embedding sequence of visual objects, visual concepts, contexts and response respectively. Note that O w is the object embedding transformed by a linear layer into the same dimension as word embedding. Turn-Level Since the dialog is multi-turn, we encode this turn order with a relative turn embedding (Bao et al., 2020). Specifically, the turn number is counted from the last utterance of the dialogue to the beginning. Note that as for the tokens corresponding to O and Q , we simply set them the same as the first utterance of C . Position-Level Positional embedding encodes the signal of the token order in the total input sequence, which is the same as positional encoding of the original transformer (Vaswani et al., 2017). Segment-Level Segment embedding is employed to differentiate which segment the token is in, i.e. , O, Q, C or R . Due to the inherent gap between visual modality and textual modality, directly optimizing the model by response generation objective may result in the insufficient utilization of the visual knowledge. To align the semantic representations of two modalities, we devise Masked Concept Prediction (MCP) objective. 15 % of the visual concepts are randomly replaced with [MASK] tokens in each training instance, which need to be predicted by the model. However, one problem still remains, i.e. , the visual concepts have no specific order when extracting from images. In other words, we need to model MCP as a matching problem of set, which does not need to consider the order of predicted concepts when there are more than two concepts masked out simultaneously. To tackle this, inspired by Hu et al. (2020), we adopt the Hungarian Matching Loss (Stewart et al., 2016; Carion et al., 2020) to estimate an optimal mapping  so that the prediction for each masked position is assigned one of the target concepts. Here we denote the set of all input as X = ( O, Q, C, R ) , the set of the bidirectional self-attention part of X as B = ( O, Q, C ) , the set of masked concepts as  Q , the set of unmasked tokens as B \  Q , and the prediction probabilities of the corresponding representations in the final layer of transformer as H = { h i } mi =1 where h i is the probability distribution of the i -th masked position. Hence, the MCP loss can be defined as: LMCP (  Q, H,  ) =  (cid:88) q  ( i )   Q log h i (cid:16) q  ( i ) | B \  Q (cid:17) (6) where  ( i ) is the index of the target concept assigned to the i -th prediction. When predicting a masked concept, the model will have to resort to visual region features, dialog contexts and other unmasked visual concepts. This would help the model to align the cross-modal representations between text and visual regions. Encouraged by the success of UniLM (Dong et al., 2019) in Seq2Seq tasks, we adopt the Masked Response Prediction (MRP) objective to model the response generation. During training, 70 % of the tokens in R are randomly masked with the special token [MASK] . The model is optimized to recover the masked tokens. The masked response tokens and other unmasked tokens in the whole input sequence can be denoted as  R and X \  R , respectively. Suppose that p i is the conditional probability distribution of the i -th token in R , the MRP loss is the Negative Log-Likelihood (NLL) of the masked Position-Level Token-Level Network Turn-Level Segment-Level Multi-Layer Transformer vis Response (  ) Dialog Context (  ) Visual Concepts (  ) Region Features (  ) vis vis vis vis vis tag tag tag tag usr usr usr usr usr usr usr usr usr usr sys sys sys sys sys sys sys sys sys sys 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 -2 -2 -2 -2 -2 -2 -2 -2 -2 -2 -2 -2 -2 -2 -2 -2 -1 -1 -1 -1 0 0 0 0 0 0 0 0 0 0 did you eat ? Note that the self-attention mask in R is left-to-right, but the rest are bidirectional. In other words, the tokens in O, Q and C can attend to each other from both directions, while the tokens in R can attend all tokens in O, Q, C and the leftward tokens in R including itself. MRP implicitly encourages the model to generate responses by learning the relationship among all input tokens. For decoding, we first encode the image regions, visual concepts, dialog contexts, and a special token [BOS] as input. Then the model starts the generation by feeding a [MASK] token and samples a word from the predicted distribution over vocabulary. Then, the [MASK] token is replaced by the generated token and a new [MASK] is appended to the input sequence for next word prediction. The generation process terminates when the model predicts [EOS] token or reaches the pre-defined maximum length. Visual Knowledge Bias Normally, the top projection layer of generation model produces a probability distribution over the vocabulary: p = softmax ( W e r + b ) , (8) where the e r  R d , W  R | V | d and b  R | V | are the last output of the transformer network, weight and bias parameters of the decoding head, respectively. | V | denotes the vocabulary size. So far, the visual world knowledge is introduced into the response generation model by the shared-parameter self-attention layers. To further inject the visual knowledge into the generation model, we design a simple but effective strategy, namely Visual Knowledge Bias (VKB). Concretely, an additional visual vocabulary bias b q is first calculated as follow: b q = F q ( e qavg ) (9) where F q : R d  R | V | is a projection layer. e qavg denotes the average pooling on all hidden representations of visual concepts, i.e. , e qavg = AvgP ooling ( E q ) where E q = ( e q 1 , ..., e qK ) . Then, we mask non-visual-concept tokens in the vocabulary and the masked vocabulary bias  b q  R | V | is added to the top layer of generation model to get the final distribution over vocabulary:  p = softmax ( W e r + b +  b q ) (10) We leverage this final vocabulary distribution to calculate the MRP loss in Eq. 7 to optimize the model. This visual knowledge bias would encourage the model to generate more visual knowledge related tokens in the response. To sum up, the final objective of our response generation model is to minimize the integrated loss: L = LMRP + LMCP (11) 5 Experimental Setup 5.1 Datasets To evaluate the performance of Maria, we conduct comprehensive experiments on the Reddit dataset released by Yang et al. (2020), which is a large-scale and high-quality multi-turn conversations extracted from Reddit Conversation Corpus (Dziri et al., 2019b). Each dialog has 3 to 5 utterances, and the training/validation/test set has 1M/20K/20K dialogs respectively. We train and validate the retrieval model using the Karpathy's split 3 of the MS-COCO image captioning data, where the images are split into 3 https://cs.stanford.edu/people/ karpathy/deepimagesent 113.2K/5K/5K samples as training/validation/test set, respectively. After the retrieval model is trained, we fetch 500K images from the Open Images dataset as the image index, and then retrieve images from it by dialog context and response to construct the training data for response generator. Both automatic metrics and human evaluation are employed to assess the performance of Maria and baselines. Automatic metrics include: (1) Fluency : perplexity (PPL) measures the confidence of the generated responses; (2) Relevance : BLEU-1 (Pa-pineni et al., 2002), Rouge-L (Lin, 2004), and we follow Serban et al. (2017) to utilize Embedding Average cosine similarity, Vector Extrema cosine similarity, and Embedding Greedy Matching score. All this metrics are calculated by running the public NLG evaluation script 4 ; (3) Diversity : Distinct-1 (Dist-1) and Distinct-2 (Dist-2) (Li et al., 2016) are defined as the number of distinct uni-grams or bi-grams divided by the total amount of words. In human evaluation, we randomly select 100 dialogue contexts and the corresponding generated responses for Maria and compared baselines. Three human annotators are asked to score the response quality on a scale of { 0, 1, 2 } from three aspects, including Fluency , Relevance and Richness . The higher score means the better. Since each response receives 3 scores on each aspect, we report the average scores over annotators and responses. The inter-annotator agreement is measured by Fleiss' Kappa(Fleiss and Cohen, 1973). For the retrieval model, ResNeXt-101-32x8d feature is used as the visual embedding, while the concatenation of the last 4 layers of BERT's outputs is used as the textual embedding. Both embeddings are then respectively fed into an MLP composed of three layers of size (1024, 1024, 512). When training the retrieval model, we set the margin M = 0 . 5 for the hinge loss, and only tune the parameters of both MLPs while freezing the parameters of ResNeXt and BERT. The total training epoch is 20. At inference, the FAISS (Johnson et al., 2019) library is utilized to accelerate the inner product search by batch processing. We use the off-the-shelf object detector from UpDown (An-derson et al., 2018) to extract top-k (k=36) image 4 https://github.com/Maluuba/nlg-eval region features and the corresponding visual concepts. The detector is a Faster R-CNN (Ren et al., 2015) model trained on the Visual Genome dataset (Krishna et al., 2017). For the response generation model, we set the number of transformer layers L = 12 and the hidden embedding dimension D = 768 . Besides, the network parameters are initialized by UniLM. The maximum sequence lengths of context and response are set to 110 and 40, respectively. The sequence lengths of region features and concept tokens are both set to 36. The batch size is 64. We use the Adam Optimizer (Kingma and Ba, 2015) with a learning rate 3e-5 to train the response generation model. The training is conducted on 4 Nvidia Tesla P40 24G GPU cards for 20 epochs. We compare the following baselines in the experiments: (1) Seq2Seq : A standard Sequence to Seqence model with attention mechanism (Bahdanau et al., 2015). (2) HRED : A Hierarchical Recurrent Encoder-Decoder neural network (Serban et al., 2016). (3) VHRED : A variation of HRED that introduces latent variables into the generation (Ser-ban et al., 2017). (4) ReCoSa : A hierarchical transformer-based model (Zhang et al., 2019) that achieves the state-of-the-art performance on benchmarks of dialog generation. (5) ImgVAE : A dialog generation model (Yang et al., 2020) that is trained on both textual dialogs and image-grounded dialogs by recovering a latent image behind the textual dialog within a conditional variational auto-encoding framework. (6) DialoGPT : An open-domain dialog model (Zhang et al., 2020) that fine-tunes GPT-2 (Radford et al., 2019) on massive Reddit data. Since DialoGPT is a dialog generation model trained on the text-only corpus, we introduce it as an auxiliary baseline. For a fair comparison, we choose the same model size ( L =12, D =768) of DialoGPT (117M) as our model. We summarize the experimental results of automatic evaluations in Table 1. Maria achieves the substantial performance improvements over baselines on all metrics except for the comparison to DialoGPT. Especially, Maria significantly surpasses ImgVAE on Dist-1/2, which indicates introducing richer visual knowledge, i.e. , image region features Model PPL BLEU-1 Rouge-L Average Extrema Greedy Dist-1 Dist-2 Seq2Seq (Bahdanau et al., 2015) 77.27 12.21 10.81 78.38 40.06 62.64 0.53 1.96 HRED (Serban et al., 2016) 84.02 11.68 11.29 75.54 37.49 60.41 0.89 3.21 VHRED (Serban et al., 2017) 78.01 12.22 11.82 75.57 39.24 62.07 0.87 3.49 ReCoSa (Zhang et al., 2019) 71.75 12.75 11.75 79.84 42.29 63.02 0.66 3.83 ImgVAE (Yang et al., 2020) 72.06 12.58 12.05 79.95 42.38 63.55 1.52 6.34 DialoGPT (Zhang et al., 2020) 36.03 5.87 5.20 77.80 35.40 58.39 10.41 49.86 Maria 54.38 14.21 13.02 82.54 44.14 65.98 8.44 33.35 Maria ( w/o MCP) 66.71 13.91 11.60 81.59 41.06 64.10 8.36 31.80 Maria ( w/o VKB) 65.51 12.76 11.76 82.49 40.22 64.49 7.15 29.44 Maria ( w/o VKB & MCP) 62.64 11.50 10.45 77.52 41.27 61.00 6.92 28.53 Maria ( w/o images) 64.75 10.70 9.15 78.89 39.88 62.39 6.88 28.01 Maria ( w/o concepts) 69.24 11.43 10.61 82.96 41.02 65.07 4.56 16.44 Maria ( w/o images & concepts) 69.50 10.75 8.34 80.62 41.15 64.25 3.69 10.11 Table 1: Evaluation results of generated responses on the test set. and the corresponding visual concepts, is beneficial to generating more diverse and informative responses. This also reflects in human evaluation of Table 2 that the richness score of Maria is higher than that of ImgVAE. Besides, in terms of relevance metrics including BLEU-1, Rouge-L, Average, Extrema and Greedy, Maria outperforms all baselines and even performs better than DialoGPT. This indicates introducing the extra visual knowledge related to dialog context can further force the model to produce more relevant responses. On the other hand, the discrepancy of data distributions between the training data ( i.e. , Image-Chat (Shuster et al., 2020) dataset) and test data ( i.e. , Reddit conversation dataset) of the text-to-image synthesis model in ImgVAE limits its performance in practice. Besides, constrained by the capability of the text-to-image synthesis model, the richness and diversity of the synthesized images are undesirable, while Maria can retrieve a variety of images from the large-scale image index. That may be the reason why ImgVAE consistently underperforms our Maria on relevance including automatic evaluation and human judgement, which also shows the superiority of the retrieval method for the zero-resource image-grounded conversation. Another observation is that Maria slightly underperforms DialoGPT on PPL and Dist-1/2. Since DialoGPT is a large-scale pre-training based dialog generation model and introduces the extra mutual information maximization objective to improve the informativeness of generated responses, which is consistent in human evaluation with respect to flu-ency and richness. We conduct extensive ablation experiments over different model variants and input components to better understand their relative importance to the dialog generation task. As shown in Table 1, training the simplified versions of Maria or removing any visual signals from input components leads to worse performance in terms of relevance and diversity. In particular, the results on the ablation study validate that: (1) The performance improvement of dialog generation benefits from the MCP's effectiveness in aligning the representations of text and vision; (2) When training Maria, introducing VKB can further improve the quality and diversity of generated responses; (3) Rich visual knowledge, i.e. , image region features and visual concepts, play a significant role in improving the performance of dialog generation. Especially, removing the visual concepts leads to a dramatic performance drop on diversity. The phenomenon is due to the lack of necessary visual concepts, Maria can not well understand the visual world knowledge when only learning from the visual features. To further investigate the quality of responses generated by Maria, we put an example of generated responses in Figure 5. As we can see from Figure 5, when the context talks about the supermarket Aldi, Maria can retrieve a pizza related image and generate the informative response grounded on Dialog Context: Maria A: No Aldi ? it, i.e. , the pizza at Aldi is the best in the world. This implies the commonsense that the supermarket usually has the pizza to sell. It is also observed that Maria pays more attention to the relevant image regions when generating the word pizza, which demonstrates that Maria could capture useful visual knowledge from the image and subsequently leverage it to generate commonsense-aware responses. More cases are demonstrated in Appendices. In this paper, we present Maria, a neural conversational agent powered by the visual world experiences. It is able to retrieve the visual world experiences with users and generate human-like responses with some visual commonsense. Extensive experiments demonstrate Maria achieves substantial improvements over the state-of-the-art methods in automatic and human evaluation. The future works could include: (1) Design a more precise and comprehensive image retriever to include multiple retrieval images; (2) Combining the retrieve module and dialog generation into an end-to-end model, instead of learning them individually; (3) Explore more efficient neural architectures to inject the visual knowledge into response generation.
abstain abstain abstain method method abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain objective abstain result result abstain abstain abstain abstain abstain objective other abstain abstain abstain abstain abstain method abstain objective abstain abstain objective objective other other other other other other method other other other abstain method method abstain abstain other abstain abstain abstain abstain abstain method abstain abstain method method method method abstain abstain method abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method other abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain other abstain other method abstain method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain other abstain method abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain method method abstain abstain other abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain other abstain abstain abstain result abstain abstain abstain abstain method abstain abstain abstain