Out of the Box: Reasoning with Graph Convolution Nets for Factual Visual Question Answering
Accurately answering a question about a given image requires combining observations with general knowledge. While this is effortless for humans, reasoning with general knowledge remains an algorithmic challenge. To advance research in this direction a novel 'fact-based' visual question answering (FVQA) task has been introduced recently along with a large set of curated facts which link two entities, i.e., two possible answers, via a relation. Given a question-image pair, deep network techniques have been employed to successively reduce the large set of facts until one of the two entities of the final remaining fact is predicted as the answer. We observe that a successive process which considers one fact at a time to form a local decision is sub-optimal. Instead, we develop an entity graph and use a graph convolutional network to 'reason' about the correct answer by jointly considering all entities. We show on the challenging FVQA dataset that this leads to an improvement in accuracy of around 7% compared to the state of the art.
This paper proposes a neural architecture for Fact Based VQA (introduced in [1]), an extension of VQA requiring leveraging of facts from an additional knowledge base. Given an image and a question about it, the model further retrieves the top-100 facts from the KB based on cosine similarity between an averaged Glove embedding representation between question and fact. Further, the facts are embedded as entities in a graph, and each entity is processed via a Graph Convolutional Network and fed to a multilayer perceptron trained to be verified as the correct answer. Results, comparisons against baselines, and ablations are presented on the FVQA dataset.  Strengths  – This is a well written and interesting work that approaches an important problem and presents a simple and effective model for the same – The approach significantly outperforms the previous state of the art on Fact based VQA, and the ablations studied and error contribution analysis validate the modeling choices effectively. In general, the experimental results are quite thorough and promising, and the qualitative results are compelling and provide good insights. – The paper provides sufficient experimental detail and the results should be possible to reproduce  Weaknesses/Questions  – In the error decomposition, the fact-retrieval error can be controlled for by setting a higher threshold at the cost of more computation. Have the authors experimented with retrieving a larger set of facts, or were diminishing returns observed after the threshold of 100? – Some additional analysis of the typical structure of the induced graph would be interesting to see – how sparse is the adjacency matrix on average? L95 states: ‘By stacking multiple layers, we are able to gather information from nodes further away’ – some analysis/empirical evidence of whether the model can pick up on longer range relationships in the graph, would also be interesting to include and would strengthen the paper.  [1] Wang, Peng, Qi Wu, Chunhua Shen, Anthony Dick, and Anton van Den Hengel. 2017. “FVQA: Fact-Based Visual Question Answering.” IEEE TPAMI  ----  I have read through the author response and reviews, and am happy to keep my accept rating. The paper is well written and presents a novel formulation and strong results for the FVQA task. The analysis and experimentation is thorough, and in my opinion the authors have provided a strong rebuttal to most concerns raised in the reviews.