Visual Dialog involves understanding the dialog history (what has been discussed previously) and the current question (what is asked), in addition to grounding information in the image, to generate the correct response. In this paper, we show that co-attention models which explicitly encode dialog history outperform models that don't, achieving state-of-the-art performance (72 % NDCG on val set). However, we also expose shortcomings of the crowd-sourcing dataset collection procedure by showing that history is indeed only required for a small amount of the data and that the current evaluation metric encourages generic replies. To that end, we propose a challenging subset (VisDialConv) of the VisDial val set and provide a benchmark of 63% NDCG. Recently, there has been an increased interest in visual dialog, i.e. dialog-based interaction grounded in visual information (Chattopadhyay et al., 2017; De Vries et al., 2017; Seo et al., 2017; Guo et al., 2018; Shekhar et al., 2018; Kottur et al., 2019; Haber et al., 2019). One of the most popular test beds is the Visual Dialog Challenge ( VisDial ) (Das et al., 2017), which involves an agent answering questions related to an image, by selecting the answer from a list of possible candidate options. According to the authors, nearly all interactions (98%) contain dialog phenomena, such as co-reference, that can only be resolved using dialog history, which makes this a distinct task from previous Visual Question Answering (VQA) challenges, e.g. (Antol et al., 2015). For example, in order to answer the question About how many? in Figure 1, we have to infer from what was previously said, that the conversation is about the skiers. In the original paper, Das et al. (2017) find that models which structurally encode dialog history, such as Memory Networks (Bordes et al., 2016) or Hierarchical Recurrent Encoders (Serban et al., 2017) improve performance. However, naive history modelling (in this case an encoder with late fusion/concatenation of current question, image and history encodings) might actually hurt performance. Massiceti et al. (2018) take this even further, claiming that VisDial can be modeled without taking history or even visual information into account. Das et al. (2019) rebutted by showing that both features are still needed to achieve state-of-the-art (SOTA) results and an appropriate evaluation procedure has to be used. In this paper, we show that competitive results on VisDial can indeed be achieved by replicating the top performing model for VQA (Yu et al., 2019b)  and effectively treating visual dialog as multiple rounds of question-answering, without taking history into account. However, we also show that these results can be significantly improved by encoding dialog history, as well as by fine-tuning on a more meaningful retrieval metric. Finally, we show that more sophisticated dialog encodings outperform naive fusion on a subset of the data which contains true dialog phenomena according to crowd-workers. In contrast to previous work on the VisDial dataset, e.g. (Kottur et al., 2018; Agarwal and Goyal, 2018; Gan et al., 2019; Guo et al., 2019; Kang et al., 2019), we are the first to conduct a principled study of dialog history encodings. Our contributions can thus be summarized as follows:  We present SOTA results on the VisDial dataset using transformer-based Modular Co-Attention (MCA) networks. We further show that models encoding dialog history outperform VQA models on this dataset. We show that curriculum fine-tuning (Bengio et al., 2009) on annotations of semantically equivalent answers further improves results. We experiment with different dialog history encodings and show that early fusion, i.e. dense interaction with visual information (ei-ther via grounding or guided attention ) works better for cases where conversational historical context is required. We release a crowd-sourced subset containing verified dialog phenomena and provide benchmark results for future research. In this section, we extend Modular Co-Attention Networks, which won the VQA challenge 2019 (Yu et al., 2019b) and adapt it to visual dialog. Different from previous co-attention networks (Kim et al., 2018; Nguyen and Okatani, 2018), MCA networks use guided attention to model dense relations between the question and image regions for better visual grounding. In the following, we explore MCA networks with different input encodings following a [model]-[input]' convention to refer to our MCA model variants; see Figure 3 for an overview. Whenever unspecified, images are represented as a bag of bottom-up features, i.e. object level representations (see Section 3). The MCA module with multi-modal fusion as depicted in Figure 2, is common to all our architectures. Inspired by the transformers (Vaswani et al., 2017), the MCA network (Yu et al., 2019b) is a modular composition of two basic attention units: self-attention and guided attention. These are arranged in an encoder-decoder composition in the MCA module (Figure 2), which performed best for VQA (Yu et al., 2019b). The Self-Attention (SA) unit in transformers (Vaswani et al., 2017) is composed of a multihead attention layer followed by a feed-forward layer. When applied to vision, the SA unit can be viewed as selecting the most relevant object-level image features for the downstream task. Specifi-cally, the scaled dot product attention takes as input key, query and value (usually same modality's embedded representations) and outputs a self-attended vector (Eq.1). Multi-head attention provides multiple representation spaces to capture different lin-guistic/grounding phenomena, which are otherwise lost by averaging using a single head. Att ( Q,K,V ) = softmax ( QKT  d K ) V MHAtt ( Q,K,V ) = Concat ( head 1 ,...head n ) WO head i = Att ( QW Qi ,KW Kk ,V W Vi ) (1) The Guided-Attention (GA) unit conditions the attention on different sequences. The key and value come from one modality, while the query comes from a different modality similar to the decoder architecture in Transformers (Vaswani et al., 2017). Similar to Eq. 1, the GA unit outputs features f i = Att ( X, Y, Y ) where X  R m  d x comes from one modality and Y  R n  d y from the other. Residual connection (He et al., 2016) and layer normalization (Ba et al., 2016) are applied to the output of both the attention and feed-forward layers similar to (Vaswani et al., 2017; Yu et al., 2019b) in both the SA and GA units. extended analogously to model the interaction between the question and history. First, the input (i.e. the question) is passed through multiple multi-head self-attention layers L , in order to get self-aware representations before acting as conditional signal to different modalities (visual or contextual history) similar to the auto-encoding procedure of Transformers. Then the final representation XL is used as the input for GA units to model cross-modal dependencies and learn the final conditioned representation YL . The learned representations XL  R m  d and YL  R n  d contain the contextualized and conditioned representations over the word and image regions, respectively. We apply attention reduction (Yu et al., 2019b) with a multi-layer perceptron (MLP) for XL (analogously for YL ). We obtain the final multi-modal fused representation z :  x = softmax ( MLP x ( XL ))  x = i = 1  m  xi x Li z = LayerNorm ( W Tx  x + W Ty  y ) (2) where  x = [  x 1 . . .  xm ]  R m are learned attention weights (same process for  y and  y ) and W x  R d  d z , W y  R d  d z are linear projection matrices (dimensions are the same for simplicity). We call this model MCA with Image component only; (MCA-I) , since it only encodes the question and image features and therefore treats each question in Visual Dialog as an independent instance of VQA, without conditioning on the historical context of the interaction. In the following, we extend the above framework to model dialog history. We experiment with late/shallow fusion of history and image (MCA-I-H), as well as modelling dense interaction between conversational history and the image representation (i.e. MCA-I-VGH, MCA-I-HGuidedQ). History guided Question (MCA-I-HGuidedQ): The network in Figure 3a is designed to model coreference resolution, which can be considered as the primary task in VisDial (Kottur et al., 2018). We first enrich the question embedding by conditioning on historical context using guided attention in the MCA module. We then use this enriched (co-reference resolved) question to model the visual interaction as described in Section 2.1. Visually grounded history with image representation (MCA-I-VGH): Instead of considering conversational history and the visual context as two different modalities, we now ground the history with the image first, see Figure 3b. This is similar in spirit to maintaining a pool of visual attention maps (Seo et al., 2017), where we argue that different questions in the conversation attend to different parts of the image. Specifically, we pass the history to attend to object-level image features using the MCA module to get visually grounded contextual history. We then embed the question to pool the relevant grounded history using another MCA module. In parallel, the question embedding is also used to ground the current visual context. At the final step, the respective current image and historical components are fused together and passed through a linear layer before decoding. Note, this model is generic enough to potentially handle multiple images in a conversation and thus could be extended for tasks e.g. conversational image editing, which is one of the target applications of visual dialog (Kim et al., 2017; Manuvinakurike et al., 2018a,b; Lin et al., 2018; El-Nouby et al., 2018). Two-stream Image and History component (MCA-I-H): Figure 3c shows the model which maintains two streams of modular co-attention networks  one for the visual modality and the other for conversational history. We follow a similar architecture for the visual component as MCA-I and duplicate the structure for handling conversational history. At the final step, we concatenate both the embeddings and pass them through a linear layer. For all the models described above, we use a discriminative decoder which computes the similarity between the fused encoding and RNN-encoded answer representations which is passed through a softmax layer to get the probability distribution over the candidate answers. We train using cross entropy over the ground truth answer: L (  ) = 1 NN = 100  n = 1 y n logP ( x n , ) (3) N denotes the number of candidate answers which is set to 100 for this task, y n is the (ground truth) label which is 0 or 1 during the training procedure, or a relevance score of the options during fine-tuning (casting it as multi-label classification). We use PyTorch 1 (Paszke et al., 2017) for our experiments 2 . Following Anderson et al. (2018), we use bottom-up features of 36 proposals from images using a Faster-RCNN (Ren et al., 2015) pre-trained on Visual Genome (Krishna et al., 2017) to get a bag of object-level 2048-d image representations. Input question and candidate options are tokenized to a maximum length of 20 while the conversational history to 200. Token embeddings in text are initialized with 300-d GloVe vectors (Penning-ton et al., 2014) and shared among all text-based encoders. The RNN encodings are implemented using LSTMs (Hochreiter and Schmidhuber, 1997). 1 https://pytorch.org/ 2 Code available at https://github.com/ shubhamagarwal92/visdial_conv We use the Adam optimizer (Kingma and Ba, 2015) both for training and fine-tuning. We use VisDial v1.0 for our experiments and evaluation. 3 The dataset contains 123K/2K/8K dialogs for train/val/test set respectively. Each dialog is crowd-sourced on a different image, consisting of 10 rounds of dialog turns, totalling approx. 1.3M turns. Each question has also been paired with a list of 100 automatically generated candidate answers which the model has to rank. To account for the fact that there can be more than one semantically correct answer (e.g. Nope, No, None, Can-not be seen), dense annotations for 2k/2k turns of train/val of the data have been provided, i.e. a crowd-sourced relevance score between 0 and 1 (1 being totally relevant) for all 100 options. As the Visual Dialog task has been posed as a ranking problem, standard information retrieval (IR) metrics are used for evaluation, such as Recall@ { 1,5,10 } to measure performance in the top N results (higher better), mean reciprocal rank (MRR) of the Ground-Truth (GT) answer (higher better), and Mean rank of the GT answer (lower better). Normalized Discounted Cumulative Gain (NDCG) is another measure of ranking quality, which is commonly used when there is more than one correct answer (provided with their relevance). Sparse Annotation Phase: We first train on sparse annotations, i.e. only 1 provided ground-truth answer, which is available for the whole training set. Here the model learns to select only one relevant answer. Curriculum Fine-tuning Phase: Dense annotations, i.e. crowd-sourced relevance weights, are provided for 0.16% of training set, which we use to fine-tune the model to select multiple semantically equivalent answers. This acts like a curriculum learning setup (Elman, 1993; Bengio et al., 2009), 3 Following the guidelines on the dataset page we report results only on v1.0, instead of v0.9. VisDial v1.0 has been consistently used for Visual Dialog Challenge 2018 and 2019. where selecting one answer using sparse annotation is an easier task and fine-tuning more difficult. 4 4.4 Baselines MCA-I-HConcQ and MCA-H: MCA-I-HConcQ is a naive approach of concatenating raw dialog history to the question while keeping the rest of the architecture the same as MCA-I. MCA-H on the other hand considers this task as only conversational (not visual) dialog with MCA module on history instead of image. RvA: We reproduce the results of Niu et al. (2019)'s Recursive Visual Attention model (RvA), which won the 2019 VisDial challenge. Their model browses the dialog history and updates the visual attention recursively until the model has suf-ficient confidence to perform visual co-reference resolution. We use their single model's open-source implementation and apply our fine-tuning procedure on the val set in Table 1. When reporting on the test set results in Table 2, we use the leaderboard scores published online which contains further unpublished enhancements based on ensembling (MReaL-BDAI). In the following, we report results on the VisDial v1.0 val set, (Table 1), as well as the test-std set, 5 (Table 2). For measuring significance (reported on p  0 . 05 ), we use Kruskal-Wallis (Kruskal and Wallis, 1952) and Wilcoxon signed rank test (Wilcoxon, 1992) with Bonferroni correction (Bon-ferroni, 1936). We report results in terms of NDCG, which is the main metric of the challenge. MCA-I-H is our best performing model. It achieves state-of-the-art performance: It outperforms the RvA baseline by almost 5 NDCG points on the val set and by over 7 points on the test set. On the official challenge test set, MCA-I-H ranks 2 nd : it improves over 7 NDCG over the best single model but loses by 2 points against a 6-strong RvA ensemble model (2019 winning entry). 4 While instance-level' curriculum learning is defined in terms of harder dialogs', in our work, we used dataset/tasklevel' curriculum finetuning. Our suggested method is a combination of curriculum learning and fine tuning (pre-training and adjusting to a specific downstream task). As such, we use the term curriculum fine-tuning' i.e. adaptation by NDCG aware curriculum during fine-tuning. 5 We only report results for our best preforming models as the number of allowed submissions to the challenge is limited. Compared to MCA-I, which treats the task as multiple rounds of VQA, encoding history improves results, but only significantly for MCA-I-VGH in the sparse annotation phase. After fine-tuning, MCA-I-VGH and MCA-I-H perform equally. MCA-I-H implements a late/shallow fusion of history and image. Architectures which model dense interaction between the conversational history and the image representations (i.e. MCA-I-VGH, MCA-I-HGuidedQ) perform comparably; only MCA-HConcQ performs significantly worse. Note that MCA-I also outperforms the baselines and current SOTA by a substantial margin (both in the sparse annotation phase and curriculum fine-tuning phase), while, counter-intuitively, there is not a significant boost by adding conversational history. This is surprising, considering that according to Das et al. (2017), 38% of questions contain a pronoun, which would suggest that these questions would require dialog history in order to be understood/grounded by the model. Furthermore, curriculum fine-tuning significantly improves performance with an average improvement of 11.7 NDCG points, but worsens performance in terms of the other metrics, which only consider a single ground truth (GT) answer. In the following, we perform a detailed error analysis, investigating the benefits of dialog history encoding encoding and the observed discrepancy between the NDCG results and the other retrieval based metrics. We performed an ablation study whereby we did not include the caption as part of historical context and compare with the results in Table 1. The performance dropped from (NDCG 72.2, MRR 42.3) to (NDCG 71.6, MRR 40.7) using our best performing MCA-I-H model after finetuning. Since the crowd-sourced conversation was based on the caption, the reduced performance was expected. In order to further verify the role of dialog history, we conduct a crowd-sourcing study to understand which questions require dialog history, in order to be understood by humans. We first test our history-encoding models on a subset (76 dialogs) of the recently released VisPro dataset (Yu et al., 2019a) which focuses on the task of Visual Pronoun Resolution. 6 Note that VisPro also contains non-referential pleonastic pronouns, i.e. pronouns used as dummy subjects when e.g. talking about the weather (Is it sunny?). We thus create a new crowd-sourced dataset 7 , which we call VisDialConv . This is a subset of the VisDial val-set consisting of 97 dialogs, where the crowd-workers identified single turns (with dense annotations) requiring historical information. In particular, we asked crowd-workers whether they could provide an answer to a question given an image, without showing them the dialog history, and select one of the categories in Table 4 (see further details in Appendix B). In order to get reliable results, we recruited 3 crowd-workers per image-question pair and only kept instances where at least 2 people agreed. Note that we only had to discharge 14.5% of the origi-6 We use the intersection of dialogs in VisDial val set and VisPro to create this subset. 7 Data collection code available at https://github. com/shubhamagarwal92/visdialconv-amt Model Sparse annotation Phase Curriculum Fine-tuning NDCG  MRR  R@1  R@5  R@10  Mean  NDCG  MRR  R@1  R@5  R@10  Mean  VisPro subset dataset MCA-I 59.80 57.88 45.39 72.24 82.76 5.84 69.82 36.2 20 54.08 70.92 10.02 MCA-I-HConcQ 61.08 61.79 48.95 77.5 86.58 4.72 68.44 38 22.24 55.79 71.71 9.17 MCA-I-HGuidedQ 61.35 60.13 47.11 75.26 86.18 5.23 68.29 36.59 21.05 53.29 70.13 9.76 MCA-I-VGH 61.68 59.33 46.18 75.53 86.71 5.07 68.97 39.21 23.68 57.11 70.53 8.83 MCA-I-H 61.72 59.62 45.92 77.11 86.45 4.85 70.87 39.8 25.39 55.13 70.39 9.42 VisDialConv (Crowd-sourced subset) dataset MCA-I 52.07 55.55 41.65 72.47 83.81 5.92 58.65 36.2 20.52 53.3 68.25 10.32 MCA-I-HConcQ 54.84 62.06 47.42 80.1 88.87 4.37 61.42 37.92 21.86 55.67 73.3 9.01 MCA-I-HGuidedQ 53.81 62.29 48.35 80.1 88.76 4.42 62.92 38.07 22.58 54.74 70.82 9.5 MCA-I-VGH 55.48 58.45 44.54 74.95 86.19 5.18 60.63 38.1 22.89 53.71 70.31 9.49 MCA-I-H 53.01 61.24 47.63 79.07 87.94 4.77 59.89 39.73 25.15 56.49 71.86 9.53 Table 3: Automatic evaluation on the subsets of VisPro and VisDialConv dataset. nal 1035 image-question pairs, leaving us with 885 examples. The results in Table 4 show that only 11% required actual dialog historical context according to the crowd-workers. Most of the time (67% cases), crowd-workers said they can answer the question correctly without requiring history. The results in Table 3 are on the subset of 97 questions which the crowd-workers identified as requiring history. 8 They show that history encoding models (MCA-I-HGuidedQ / MCA-I-HConcQ / MCA-I-H / MCA-I-VGH) significantly outperform MCA-I, suggesting that this data cannot be modelled as multiple rounds of VQA. It can also be seen that all the models with dense (early) interaction of the historical context outperform the one with late interaction (MCA-I-H) in terms of NDCG. Models with dense interactions appear to be more reliable in choosing other correct relevant answers because of the dialog context. 8 We took care to only include examples from Visdial val set in both Vispro and VisDialConv subsets. Also note, there are only 8 overlapping instances between Vispro and VisdialConv subsets. Our best performing model on VisDialConv is MCA-I-HGuidedQ and achieves a NDCG value of 62.9 after curriculum fine-tuning. However, on the VisPro subset, we observe that MCA-I-H still outperforms the other models. Interestingly, on this set, MCA-I also outperforms other history encoding models (except for MCA-I-H). In sum, our analysis shows that only a small subset of the VisDial dataset contains questions which require dialog history, and for those, models which encode history lead to better results. We posit that this is due to the fact that questions with pleonastic pronouns such as Is it sunny/daytime/day. . .  are the most frequent according to our detailed analysis in Appendix C about the dialog phenomena. Here, we investigate the discrepancy between the NDCG results and the other retrieval-based methods. First, we find that the annotation scales differs: while there is a 3-way annotation on the train set, the val set defines 6 possible relevance classes, see Table 5. This affects the evaluation results of our Image Dialog MCA-I-H MCA-I-VGHA bag of chips and a apple and orange. Next, a manual inspection reveals that the relevance weight annotations contain substantial noise: We find that ground truth answers were marked as irrelevant for about 20% of train and 10% of val set. Thus, our models seem to get confused by fine-tuning on this data. We, therefore, manually corrected the relevance of only these GT answers (in dense annotations of train set only, but not in val set). Please see Appendix D for further details. The results in Table 1 (for MCA-I-H-GT) show that the model fine-tuned on the corrected data still achieves a comparable NDCG result, but substantially improves stricter (single answer) metrics, which confirms our hypothesis. Finally, due to the noisy signal they receive during fine-tuning, our models learn to select safe answers 9 , such as I can't tell (see examples in 9 We show the statistics of top-ranked predictions by our MCA-I-H model on our VisdialConv subset (i.e. 97 dialogs of the Visdial val set). Read as: (Response, count, %) (Yes, 14, 14%) (No, 11, 11.34%) (I cannot tell, 9, 9.27%) (Nope, 3, 3%) (Not that I see, 2, 2.06%) (Red and white, 2, 2.06%) (Not sure, 2, 2.06%) (I can't tell, 2, 2.06%). This shows that Figure 4), which rank high according to (the more forgiving) NDCG, but perform poorly for stricter metrics like MRR and Recall. Our results suggest that the VisDial dataset only contains very limited examples which require dialog history. Other visual dialog tasks, such as GuessWhich? (Chattopadhyay et al., 2017) and GuessWhat?! (De Vries et al., 2017) take place in a goal-oriented setting, which according to Schlangen (2019), will lead to data containing more natural dialog phenomena. However, there is very limited evidence that dialog history indeed matters for these tasks (Yang et al., 2019). As such, we see data collection to capture visual dialog phenomena as an open problem. Nevertheless, our results also show that encoding dialog history still leads to improved results. This is in contrast with early findings that a) naive encoding will harm performance (Das et al. (2017); at least 13.3% of answers are non-commital (I cannot tell, Not sure, I can't tell). see MCA-I-HConcQ in Table 1), or that b) history is not necessary (Massiceti et al., 2018). Furthermore, we find that our model learns to provide generic answers by taking advantage of the NDCG evaluation metric. Learning generic answers is a well-known problem for open-domain dialog systems, e.g. (Li et al., 2016). While the dialog community approaches these phenomena by e.g. learning better models of coherence (Xu et al., 2018), we believe that evaluation metrics also need to be improved for this task, as widely discussed for other generation tasks, e.g. (Liu et al., 2016; Novikova et al., 2017; Reiter, 2018). As a first step, BERT score (Zhang et al., 2019) could be explored to measure ground-truth similarity replacing the noisy NDCG annotations of semantic equivalence. In sum, this paper shows that we can get SOTA performance on the VisDial task by using transformer-based models with Guided-Attention (Yu et al., 2019b), and by encoding dialog history and fine-tuning we can improve results even more. Of course, we expect pre-trained visual BERT models to show even more improvements on this task, e.g. Vilbert (Lu et al., 2019), LXMert (Tan and Bansal, 2019), UNITER (Chen et al., 2019) etc. However, we also show the limitations of this shared task in terms of dialog phenomena and evaluation metrics. We, thus, argue that progress needs to be carefully measured by posing the right task in terms of dataset and evaluation procedure. We thank the anonymous reviewers for their insightful comments. Shubham would like to thank Raghav Goyal for the discussions during Pikabot' submission to Visual Dialog Challenge 2018. This work received continued support by Adobe Research gift funding for further collaboration. This research also received funding from Adeptmind Inc., Toronto, Canada and the EPSRC project MaDrIgAL (EP/N017536/1). We would also like to acknowledge the AWS Cloud Credits for Research programme.
abstain result result objective abstain abstain abstain abstain abstain abstain abstain abstain result result result abstain objective result result result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method other method abstain method abstain abstain method abstain abstain method method abstain abstain abstain abstain method abstain method abstain method method method abstain abstain abstain other method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method other other abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result other other other other other other other other result result other other other other other result other result other result result result method other other other other other