Chain of Reasoning for Visual Question Answering
Reasoning plays an essential role in Visual Question Answering (VQA). Multi-step and dynamic reasoning is often necessary for answering complex questions. For example, a question "What is placed next to the bus on the right of the picture?" talks about a compound object "bus on the right," which is generated by the relation <bus, on the right of, picture>. Furthermore, a new relation including this compound object <sign, next to, bus on the right> is then required to infer the answer. However, previous methods support either one-step or static reasoning, without updating relations or generating compound objects. This paper proposes a novel reasoning model for addressing these problems. A chain of reasoning (CoR) is constructed for supporting multi-step and dynamic reasoning on changed relations and objects. In detail, iteratively, the relational reasoning operations form new relations between objects, and the object refining operations generate new compound objects from relations. We achieve new state-of-the-art results on four publicly available datasets. The visualization of the chain of reasoning illustrates the progress that the CoR generates new compound objects that lead to the answer of the question step by step.
*Summary* Questions in Visual Question Answering often require to reason about referring expressions that relate some object to other objects in the image, e.g. “The bus on the right of the person that is wearing a hat”. This paper argues that previous architectures for VQA are not explicitly biased to capture such chains of reasoning and propose a novel architecture that interleaves a relational reasoning step (to form novel relationships between objects) with a refinement operator (to prevent exponential blow-up). The model improves the state-of-the-art on four VQA datasets:  VQA v1 and v2, Coco QA and TDIUC. In the ablation studies, the authors show that the chain of reasoning structure outperforms the stack-based and parallel-based attention structure.    *Originality* The observation that referring expressions are compositional and that neural networks for VQA should incorporate compositional priors is not novel (see e.g. compositional attention network [Hudson and Manning, ICLR2018]). Nevertheless, it is interesting that this paper applies a relational prior on top of extracted *object* features, as opposed to prior work that has proposed to apply it on top of convnet features [Relation Networks, Santoro et al]. Also, the relational reasoning module is different from prior work, and these details seem to matter; The ablation study highlights the benefit of the proposed module over a vanilla relation network module.  *Clarity* The paper is well-structured but could benefit from an additional pass of proofreading. The model description is mathematically precise, and I believe one could reproduce the experimental results from the paper description.   *Quality* Experiments are thoroughly executed, improving the state-of-the-art on four VQA datasets. Also, the ablation study compares the relative improvements of several hyperparameters of the proposed module, as well as relative improvements over related architectures.  *Significance* The proposed model is strong performant, and could be widely applied to a range of language-vision tasks.   Some of my remaining concerns: - Although evaluation is thorough, I believe some other datasets would be better suited to evaluate the proposed model. Instead of Coco QA, I would have suggested evaluating on CLEVR [Johnson et al, CVPR17] or the oracle task of GuessWhat [de Vries, CVPR17].  - In the ablation studies, the authors mention that the chain of reasoning structure outperforms the stacked attention baseline and parallel attention baseline. Although I remember the stacked attention network architecture by heart, I did not figure out what the parallel architecture would be, so maybe you can add a few lines to that paper to make it more self-contained?  Post-rebuttal Feedback I'm keeping my scores.