Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
We marry two powerful ideas: deep representation learning for visual recognition and language understanding, and symbolic program execution for reasoning. Our neural-symbolic visual question answering (NS-VQA) system first recovers a structural scene representation from the image and a program trace from the question. It then executes the program on the scene representation to obtain an answer. Incorporating symbolic structure as prior knowledge offers three unique advantages. First, executing programs on a symbolic space is more robust to long program traces; our model can solve complex reasoning tasks better, achieving an accuracy of 99.8% on the CLEVR dataset. Second, the model is more data-and memory-efficient: it performs well after learning on a small number of training data; it can also encode an image into a compact representation, requiring less storage than existing methods for offline question answering. Third, symbolic program execution offers full transparency to the reasoning process; we are thus able to interpret and diagnose each execution step.
The paper studies the combination of deep representation learning for visual recognition and language understanding, and symbolic program execution for reasoning.  The focus is on the development of a Visual Question Answering System, where question related to a scene are posted to the system. The idea is quite simple to understand: the system first recovers information related to the structure of the scene and represents it symbolically, together with the symbolic representation (in the form of a program) of the question. Then, the program is executed over the representation of the scene and the answer is computed.  The first step is done by means of machine learning techniques for visual recognition and NLP techniques (when the question is posed in NL) to obtain the program associated to the question. The learned knowledge is then represented symbolically which provides the ability of a higher level of reasoning about the scene. The authors then show experimentally that the proposed system has a high percentage of accuracy on a benchmark dataset. They also show that it is more efficient in data and memory usage (the symbolic abstraction helps in reasoning and learning with fewer examples, as humans do). The authors state that to the best of their knowledge this is the first work that makes this combination allowing also for transparency and explanations in the reasoning process. To the best of my knowledge this true, but as this is not my area of expertise cannot be completely sure.  The idea is novel, is timely, and sound to the extent I checked. The authors hypothesis is that machine learning techniques is not enough to perform high level reasoning tasks (as question answering) over visual scenes. A way to incorporate and structure context knowledge is necessary. This proposal achieves exactly that.  My only concern is that the representation language used is not clearly defined. There is no formal definition or specification of what is a symbolic program. What language is used? What is a symbolic module? I this sense I find the work still behind of what symbolic reasoning is supposed to involve.  Author's Feedback: the authors have answered satisfactorily my concerns.