Visual Question Answering with Question Representation Update (QRU)
Our method aims at reasoning over natural language questions and visual images. Given a natural language question about an image, our model updates the question representation iteratively by selecting image regions relevant to the query and learns to give the correct answer. Our model contains several reasoning layers, exploiting complex visual relations in the visual question answering (VQA) task. The proposed network is end-to-end trainable through back-propagation, where its weights are initialized using pre-trained convolutional neural network (CNN) and gated recurrent unit (GRU). Our method is evaluated on challenging datasets of COCO-QA [19] and VQA [2] and yields state-of-the-art performance.
This paper studies the task of visual question answering. In order to answer questions about the images, this work derives their model from neural reasoners. Neural reasoners can take a question, and supporting facts necessary to answer the question and produce updated query based on the refinement provided by the supporting facts. This task proposes to use object detection (with their spatial position) as facts necessary to answer the question. The results on standard test datasets for VQA show some promise but do not match current state of the art methods performance.
The paper is well written and the experiments appear sound. The overall motivation of having more directed and detailed question rather than a vague one is also important for VQA. The method is derived from neural reasoning system previously developed for text based QA. The authors do not offer any improvements or modifications to the reasoning system but rather use objects in the image as facts instead of textual facts in traditional neural reasoning system. Hence, the overall novelty of the proposed method is minor.   Additionally, I am not fully convinced that object detection boxes alone provide enough facts necessary for answering the questions in the current datasets. Firstly, such method invariably perform poorly for counting questions which the authors also make note of (line 220). Second, the object and their locations should only be helpful for questions directly related to objects. The authors contend that 'most of the questions are related to objects' (line 97-99), but that is not the case for the VQA dataset. Many questions refer to object but are actually seeking information near the object (What is next to the couch? What is to the left of the dog? etc..). Moreover, the object proposals are not drawn based on the entities referred in the question but rather by using automated 'top-ranked' regions. So, the use of objects in the scene (even when used in conjunction with their location) serves only as limited source of knowledge. Since the entire set of 'facts' also include the whole image, it is not possible to study the effects of including the edge boxes. The authors do not present ablations studies to prove that the inclusion of facts do indeed help the reasoning system. It could be done by training a controlled system devoid of 'overall' image feature and only consisting of the object proposals.   On the same token, the attention map is also produced on the top-ranked image regions that are generated without knowledge of the question. A grid-layout (often obtained by CNN networks before pooling layers [1] [2] etc..) is also not used which means that certain image regions (that are not bounded by the top-ranked regions) are unavailable for attending. The resolution provided by including the whole image does not provide granularity required for attending to specific parts.  Similarly, the authors show a number of examples of question representation update where the updated question, at times, show greater degree of detail in the question. However, I am not convinced that this can be taken as indication of successful representation update for several reasons. 1) The authors use nearest neighbor approach to find the closest question in the dataset that match the updated question representation. It is unsurprising that the updated representations contain references to objects present in the image since the input to the reasoning system consist of explicit object locations and features from them. It is, however, uncertain whether the updated representation actually asks about the same entity as the original question. Certainly, the updated question contain more detail but is the meaning of the question unchanged to the original? 2) What do the failure cases look like? 3) The examples only include questions from COCO-QA dataset, which are derived from captions via NLP algorithms and are notoriously bad in terms of grammar, composition and phrasing.   The above comment does not, however, mean that the updated representation wont produce better query. This only means that the proposed nearest neighbor method to assess or demonstrate method's efficacy is dubious.  In a nutshell, the paper is well-written and presents a interesting method of using reasoning engine for VQA. However, the authors do not offer any new innovation on the neural reasoning and the image based fact generation proposed by the authors may be inadequate to completely reason about the images. The general applicability of the work in current form is low, especially considering similar methods offer better performance in VQA. For a stronger case, the visual facts need to be question specific and more detailed so that it can offer better basis for reasoning. The explicit efficacy of the proposed work also should be demonstrated by the help of ablation studies.   ----------- References ----------- 1.Kevin J Shih, Saurabh Singh, and Derek Hoiem. Where to look: Focus regions for visual question answering. In CVPR, 2016.  2.Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In CVPR, 2016.