Visual Reference Resolution using Attention Memory for Visual Dialog
Visual dialog is a task of answering a series of inter-dependent questions given an input image, and often requires to resolve visual references among the questions. This problem is different from visual question answering (VQA), which relies on spatial attention (a.k.a. visual grounding) estimated from an image and question pair. We propose a novel attention mechanism that exploits visual attentions in the past to resolve the current reference in the visual dialog scenario. The proposed model is equipped with an associative attention memory storing a sequence of previous (attention, key) pairs. From this memory, the model retrieves the previous attention, taking into account recency, which is most relevant for the current question, in order to resolve potentially ambiguous references. The model then merges the retrieved attention with a tentative one to obtain the final attention for the current question; specifically, we use dynamic parameter prediction to combine the two attentions conditioned on the question. Through extensive experiments on a new synthetic visual dialog dataset, we show that our model significantly outperforms the state-of-the-art (by ≈ 16 % points) in situations, where visual reference resolution plays an important role. Moreover, the proposed model achieves superior performance (≈ 2 % points improvement) in the Visual Dialog dataset [1], despite having significantly fewer parameters than the baselines.
This paper proposed a visual reference resolution model for visual dialog. The authors proposed to attentions, 1: tentative attention that only consider current question and history, and 2: relevant attention that retrieved from an associate attention memory. Two attentions are further combined with a dynamic parameter layer from [9] and predict the final attention on image. The authors create MNIST Dialog synthetic dataset which model the visual reference resolution of ambiguous expressions. The proposed method outperform baseline with large margin. The authors also perform experiments on visual dialog dataset, and show improvements over the previous methods. 
[Paper Strengths]
1: Visual reference resolution is a nice and intuitive idea on visual dialog dataset. 
2: MNIST Dialog synthetic dataset is a plus.
3: Paper is well written and easy to follow. 
[Paper Weaknesses]
My major concerns about this paper is the experiment on visual dialog dataset. The authors only show the proposed model's performance on discriminative setting without any ablation studies.  There is not enough experiment result to show how the proposed model works on the real dataset. If possible, please answer my following questions in the rebuttal. 
1: The authors claim their model can achieve superior performance having significantly fewer parameters than baseline [1]. This is mainly achieved by using a much smaller word embedding size and LSTM size. To me, it could be authors in [1] just test model with standard parameter setting. To backup this claim, is there any improvements when the proposed model use larger word embedding, and LSTM parameters? 
2: There are two test settings in visual dialog, while the Table 1 only shows the result on discriminative setting. It's known that discriminative setting can not apply on real applications, what is the result on generative setting? 
3: To further backup the proposed visual reference resolution model works in real dataset, please also conduct ablation study on visDial dataset. One experiment I'm really interested is the performance of ATT(+H) (in figure 4 left). What is the result if the proposed model didn't consider the relevant attention retrieval from the attention memory.