Visual Dialog is a multimodal task of answering a sequence of questions grounded in an image (using the conversation history as context). It entails challenges in vision, language, reasoning, and grounding. However, studying these subtasks in isolation on large, real datasets is infeasible as it requires prohibitively-expensive complete annotation of the state' of all images and dialogs. We develop CLEVR-Dialog, a large diagnostic dataset for studying multi-round reasoning in visual dialog. Specifically, we construct a dialog grammar that is grounded in the scene graphs of the images from the CLEVR dataset. This combination results in a dataset where all aspects of the visual dialog are fully annotated. In total, CLEVR-Dialog contains 5 instances of 10-round dialogs for about 85 k CLEVR images, totaling to 4 . 25 M question-answer pairs. We use CLEVR-Dialog to benchmark performance of standard visual dialog models; in particular, on visual coreference resolution (as a function of the coreference distance). This is the first analysis of its kind for visual dialog models that was not possible without this dataset. We hope the findings from CLEVR-Dialog will help inform the development of future models for visual dialog. Our code and dataset are publicly available 1 . The focus of this work is on intelligent systems that can see (perceive their surroundings through vision), talk (hold a visually grounded dialog), and reason (store entities in memory as a dialog progresses, refer back to them as appropriate, count, compare, etc .). Recent works have begun studying such systems under the umbrella of Visual Dialog (Das et al., 2017a; de Vries et al., 2017), where 1 https://github.com/satwikkottur/clevr-dialog an agent must answer a sequence of questions grounded in an image. As seen in Fig. 1, this entails challenges in  vision ( e.g ., identifying objects and their attributes in the image), language/reasoning ( e.g ., keeping track of and referencing previous conversation via memory), and grounding ( e.g ., grounding textual entities in the image). In order to train and evaluate agents for Visual Dialog, Das et al. (2017a) collected a large dataset of human-human dialog on real images collected between pairs of workers on Amazon Mechanical Turk (AMT). While such large-scale realistic datasets enable new lines of research, it is difficult to study the different challenges (vision, language, reasoning, grounding) in isolation or to break down the performance of systems over different challenges to identify bottlenecks, because that would require prohibitively-expensive complete annotation of the state' of all images and dialogs (all entities, coreferences, etc .). In this work, we draw inspiration from Johnson et al. (2017), and develop a large diagnostic datasetCLEVR-Dialogfor studying and benchmarking multi-round reasoning in visually-grounded dialog. Each CLEVR image is synthetically rendered by a particular scene graph (Johnson et al., 2017) and thus, is by construction exhaustively annotated. We construct a dialog grammar that is grounded in these scene graphs. Specifically, similar to Das et al. (2017b), we view dialog generation as communication between an Answerer (A-er) who can see' the image and has the complete scene graph (say S a ), and a Questioner (Q-er), who does not see' the image and is trying to reconstruct the scene graph over rounds of dialog (say S tq ). As illustrated in Fig. 1, the dialog begins by A-er providing a grounded caption for the image, which conveys some but not all information about S a . The Q-er builds a partial scene graph S 0 q based on the caption, and follows up by asking questions Figure 1: CLEVR-Dialog: we view dialog generation as communication between an Answerer (A-er) who can see' the image I and has the complete scene graph S a (far right), and a Questioner (Q-er), who does not see' the image. grounded in S 0 q , which the A-er answers, and the dialog progresses. Our dialog grammar defines rules and templates for constructing this grounded dialog. Note that A-er with access to S a (perfect vision) exists only during dialog generation to obtain ground truth answers. While studying visual dialog on CLEVR-Dialog, models are forced to answer questions with just the image and dialog history as additional inputs. In total, CLEVR-Dialog contains 5 instances of 10-round dialogs for each of 70 k (train) and 15 k (val) CLEVR images, totaling to 3 . 5 M (train) and 0 . 75 M (val) question-answer pairs. We benchmark several visual dialog models on CLEVR-Dialog as strong baselines for future work. The combination of CLEVR images (with full scene graph annotations) and our dialog grammar results in a dataset where all aspects of the visual dialog are fully annotated. We use this to study one particularly difficult challenge in multi-dialog visual reasoning  of visual coreference resolution . A coreference arises when two or more phrases ( coreferring phrases ) in the conversation refer to the same entity ( referent ) in the image. For instance, in the question What about that cylinder?' (Q3) from Fig. 1, the referent for the phrase that cylin-der' can be inferred only after resolving the phrase correctly based on the dialog history, as there are multiple cylinders in the image. We use CLEVR-Dialog to diagnose performance of different methods as a function of the history dependency ( e.g ., coreference distancethe number of rounds between successive mentions of the same object) and find that the performance of a state-of-art model (CorefNMN) is at least 30 points inferior for questions involving coreference resolution compared to those which do not (Fig. 5), highlighting the challenging nature of our dataset. This is the first analysis of its kind for visual dialog that was simply not possible without this dataset. We hope the findings from CLEVR-Dialog will help inform the development of future models for visual dialog. Coreference Resolution is a well studied problem in the NLP community (Ng, 2010; Wiseman et al., 2016; Lee et al., 2017; Clark and Manning, 2016a,b). Our work focuses on visual coreference resolution  the referent is now a visual entity to be grounded in visual data. Several works have tackled visual coreference resolution in videos (Ra-manathan et al., 2014; Rohrbach et al., 2017) and 3D data (Kong et al., 2014), and have introduced real image datasets for the same (Hodosh et al., 2014). Visual Dialog and Synthetic Datasets. We contrast CLEVR-Dialog against four existing datasets: (1) CLEVR (Johnson et al., 2017) is a diagnostic dataset for visual question answering (VQA) (An-tol et al., 2015) on rendered images that contain objects like cylinders, cubes, etc ., against a plain background (Fig. 1). While CLEVR-Dialog uses the same set of images, the key difference is that of focus and emphasis  the objective of CLEVR-Figure 2: Example dialogs from MNIST Dialog, CLEVR-Dialog, and VisDial, with coreference chains manually marked for VisDial and automatically extracted for MNIST Dialog and CLEVR-Dialog. VQA questions is to stress-test spatial reasoning in independent single-shot question answering; the objective of CLEVR-Dialog is to stress-test temporal or multi-round reasoning over the dialog history. (2) CLEVR-Ref+ (Liu et al., 2019) is a diagnostic dataset based on CLEVR images for visual reasoning in referring expressions. CLEVR-Dialog goes beyond CLEVR-Ref+, which focuses on grounding objects given a natural language expression, and deals with additional visual and linguistic challenges that require multi-round reasoning in visual dialog. (3) MNIST-Dialog (Seo et al., 2017) is a synthetic dialog dataset on a grid of 4  4 stylized MNIST digits (Fig. 2). While MNIST-Dialog is similar in spirit to CLEVR-Dialog, key difference is complexity  the distance between a coreferring phrase and its antecedent is always 1 in MNIST-Dialog; in contrast, CLEVR-Dialog has a distribution ranging from 1 to 10. (4) VisDial (Das et al., 2017a) is a large scale visual dialog dataset collected by pairing two human annotators (a Q-er and an A-er) on AMT, built on COCO (Lin et al., 2014) images. VisDial being a large open-ended real dataset encompasses all the challenges of visual dialog, making it difficult to study and benchmark progress on individual challenges in isolation. Fig. 2 qualitatively compares MNIST-Dialog, CLEVR-Dialog, and VisDial, and shows coreference chains (manually annotated for this VisDial example, and automatically computed for MNIST-Dialog and CLEVR-Dialog). We can see that the chains in MNIST-Dialog are the simplest (distance always 1). While coreferences in VisDial can be on a similar level of difficulty than CLEVR-Name CLEVR MNIST VisDial Dialog (ours) Dialog # Images 85 k 50 k 123 k # Dialogs 425 k 150 k 123 k # Questions 4 . In this section, we describe the existing annotation for CLEVR images, then detail the generation process for CLEVR-Dialog, and present the dataset statistics in comparison to existing datasets. Setup. Every CLEVR image I has a full scene graph annotation, S a . This contains information about all the objects in the scene, including four major attributes { color, shape, material, size } , 2D image and 3D world positions, and relationships { front, back, right, left } between these objects. We only use objects, attributes, and relationships. Dialog Grammar. An important characteristic of visual dialog that makes it suitable for practical applications is that the questioner does not see' the image (because if it did, it would not need to ask questions). To mimic this setup, we condition our question generation at round t only on the partial scene graph S tq that accumulates information received so far from the dialog history (and not on S a ). Specifically, we use a set of caption { T Ci } and question { T Qi } templates, which serve as the structural units of our dialog grammar. The role of the caption is to seed the dialog and initialize S 0 q . Each of the question templates is accompanied by a set of constraints on S tq , which decide if a particular template can be selected at the current round. For instance, a question What shape is the blue object?' can be only be asked (generated) if the dialog so far has already mentioned a blue object', i.e ., only if S tq contains a (unique) blue object'. The nature and difficulty of the dataset is highly dependent on these templates, thus making their selection crucial. To this end, we carefully design four categories of caption templates: (a) Obj-unique mentions an object with unique set of attributes in the image, (b) Obj-count specifies the presence of a group of objects with common attributes, (c) Obj-extreme describes an object at one of the positional extremes of the image (right, left, fore, rear, center), (d) Obj-relation talks about the relationship between two objects along with their attributes in a way that allows them to be uniquely identified in the complete scene graph S a . For the questions, we experiment with three different categories: (a) Count questions ask for a count of objects in the image satisfying specific conditions, e.g ., How many objects share the same color as this one?' , (b) Existence questions are yes/no binary questions that verify conditions in the image, e.g ., Are there any other cubes?' , and (c) Seek questions query attributes of objects, e.g ., What color is that cylinder?' . Note that CLEVR-Dialog represents not just a static dataset but also a recipe for constructing increasingly challenging grounded dialog by expanding this grammar. Refer to the appendix for further details. Dialog Generation. At a high level, dialog generation now simply' involves selecting a sequence of templates such that the accompanying constraints are satisfied by S t q at all t . As a tractable approximation to this exponentially-large constraint satisfaction problem, we use beam search that finds a valid solution and enforces additional conditions to make the dialog interesting (see Fig. 4). At every round of the dialog (after 3 rounds), we ensure that each of the question template typescount, existence, and seekfalls within a range (10%  20% for count/existence each, and 30%  60% for seek). In addition, we identify independent questions that do not need history to answer them, e.g ., How many objects are present in the image?' , and limit their number to under 10%. We found this to be effective both in terms of speed and dialog diversity. Fig. 4 illustrates the diverse set of candidate questions generated at each round for a given image. Dataset Statistics. We compare CLEVR-Dialog to MNIST-Dialog and VisDial in Tab. 1, but the key measure of coreference distance cannot be reported for VisDial as it is not annotated. Overall, CLEVR-Dialog has 3  the questions and a striking 206  the unique number of questions than MNIST-Dialog , indicating higher linguistic diversity. CLEVR-Dialog questions are longer with a mean length of 10 . 6 compared to 8 . 9 for MNIST-Dialog. Crucially, supporting our motivation, the mean distance (in terms of rounds) between the coreferring expressions in CLEVR-Dialog is 3 . 2  compared to 1 . 0 in MNIST-Dialog. Moreover, the distances (see Fig. 3b) in CLEVR-Dialog vary (min of 1, max of 10), while it is constant (at 1) in MNIST-Dialog, making it easy for models to pick Figure 4: Dialog generation in CLEVR-Dialog. up on this bias. The distribution of caption and question templates is given in Fig. 3a. See appendix for further analysis. Baselines. To benchmark performance, we evaluate several models on CLEVR-Dialog. Random picks an answer at random. Random-Q picks an answer at random among valid answers for a given question type ( e.g ., name of a color for color ques-tions). Further, we adapt the discriminative visual dialog models from Das et al. (2017a): (a) L ate F usion ( LF ) that models separately encode each of question (Q), history (H), and image (I); and then fuse them by concatenation. (b) H ierarchical R ecurrent E ncoder ( HRE ) that models dialog via both dialog-level and sentence-level recurrent neural networks. (c) M emory N etwork ( MN ) that stores history as memory units and retrieves them based on the current question. We also consider neural modular architectures: (a) CorefNMN (Kot-tur et al., 2018) that explicitly models coreferences in visual dialog by identifying the reference in the question (textual grounding) and then localizing the referent in the image (visual grounding), (b) NMN (Hu et al., 2017), which is a history-agnostic ablation of CorefNMN. Results. We use multi-class classification accuracy for evaluation since CLEVR-Dialog has one-word answers. Tab. 2 shows the performance of different models. The key observations are: (a) Neural models outperform random baselines by a large margin. The best performing model, CorefNMN, outperforms Random-Q by 35%. (b) History-agnostic models (LF-Q, LF-QI, NMN) also suffer in performance, highlighting the importance of history. (c) Finally, we break down the performance of top-3 models on questions which depend on entire history ( All ), require coreference resolution ( Coref ), and are history-independent ( None ), in Fig. 5. We find that CorefNMN is 30% worse on Coref than None questions, signifying the complexity of CLEVR-Dialog as the former are qualitatively harder to answer than the latter. (d) More interestingly, HRE-QIH, though inferior to CorefNMN on Coref , outperforms the latter on All questions ( How many other objects?' ) by around 20%. A possible explanation is that the former, owing to its dialog-level RNN, captures global summaries more efficiently than the latter. This is the first analysis of its kind for visual dialog that was simply not possible without this dataset. Appendix provides a further analysis of model performances. Conclusion. We proposed a large, synthetic dataset called CLEVR-Dialog, to study multi-round reasoning in visual dialog, and in particular the challenge of visual coreference resolution. We benchmarked several qualitatively different models from prior work on this dataset, which act as baselines for future work. Our dataset opens the door to evaluate how well models do on visual coreference resolution, without the need to collect expensive annotations on real datasets.
abstain abstain abstain objective method abstain abstain abstain method abstain abstain abstain result other abstain abstain abstain abstain abstain method result abstain result abstain method abstain abstain abstain abstain abstain abstain result method abstain abstain result abstain abstain other method other other abstain other other other other other other other other other other abstain other method abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain