Hierarchical Question-Image Co-Attention for Visual Question Answering
A number of recent works have proposed attention models for Visual Question Answering (VQA) that generate spatial maps highlighting image regions relevant to answering the question. In this paper, we argue that in addition to modeling "where to look" or visual attention, it is equally important to model "what words to listen to" or question attention. We present a novel co-attention model for VQA that jointly reasons about image and question attention. In addition, our model reasons about the question (and consequently the image via the co-attention mechanism) in a hierarchical fashion via a novel 1-dimensional convolution neural networks (CNN). Our model improves the state-of-the-art on the VQA dataset from 60.3% to 60.5%, and from 61.6% to 63.3% on the COCO-QA dataset. By using ResNet, the performance is further improved to 62.1% for VQA and 65.4% for COCO-QA.
1 .
This paper presents a new method for VQA. As a main novelty, the paper proposes a co-attention mechanism that jointly reasons about visual (image) and textual (question) attentions. Furthermore, the proposed method considers a hierarchical model by using co-attention maps at a word, phrase, and question levels. Results indicate a slightly improvement with respect to alternative state-of-the-art approaches.   
The paper presents an incremental contribution with respect to previous methods for VQA that only exploit an image attention mechanism guided by question data. Here, they also consider a question attention mechanism guided by image information. In this sense, the main hypothesis of this work is that jointly considering visual and question attention mechanisms can improve the performance of current VQA systems. I agree that this hypothesis can be relevant for the case of long questions, but I believe there is also a risk that question based attention guided by image information can be misleading, in the sense that usually an image includes several information sources, while the question is more focused. In Figure 3, authors include a graph that shows the impact of question length in performance, while this figure seems to show a tendency, the effect is still weak, maybe a numerical analysis can help to support this point. I believe, an analysis of potential differences (not only question length) between most common errors of previous works (only image attention) and the proposed approach (image and question attention) can help to support the relevance of the proposed attention mechanism.  In terms of numerical results, in the abstract and in Section 4.3, the authors claim an increase in performance with respect to alternative state-of-the-art approaches of 60.4% to 62.1% in VQA dataset, and 61.6% to 65.4% in COCO-QA. However, these numbers are misleading, they consider different visual features. In particular, alternative methods use VGG, while the proposed method uses deep residual features which provide superior performance. Considering the same VGG features, the proposed method presents a smaller increase 60.3% to 60.5% in VQA and 61.6% to 63.3% in COCO-QA. While the authors also include results of their method using VGG features, it is important to clarify this part, for example, in the abstract.  Authors claim novelty in the CNN convolucional 1D method for text processing, but I believe the proposed method include just minor variations with respect to previous works, such as https://arxiv.org/pdf/1511.02274v2.pdf.   In some parts of the paper, the description of the method is very limited. I believe, it will be difficult to repeat the results of this paper based only on the information included. As an example, it is not clear to me, how do the method combine the hierarchical representations for the question embedding (word, phrase, question)?, is it the same approach used to encode the answers (Eq.7)?. Also, in Eq. 6, why do you use the same projection matrix for visual and question features?  The use of Eq.4 instead of Eq.3 to obtain the attention maps needs further explanation. It will be good to include comparative results. Also, it will be good to include more intuitions about the different terms in Eq. 4.   In Figure 4, the color attention code for words is not clear. In this sense, the visualization in Figure 4 does not support the claim: "our model is capable of localizing the main part of the question". I recommend to normalize the values to highlight with red the most relevant parts of the question.   Minor issues: - Line 107, what the "c" stands for? - Line 88 hierarchical - Line 124, c_{i,j} should be C.  - Eq. 5, i=1..N instead of n=1..N. - Line 142, equal to 1.  - Line 150, missing a "to".  - Line 136 Breifly. - Line 237 "we can find some have"