We propose a new class of probabilistic neural-symbolic models for visual question answering (VQA) that provide interpretable explanations of their decision making in the form of programs, given a small annotated set of human programs. The key idea of our approach is to learn a rich latent space which effectively propagates program annotations from known questions to novel questions. We do this by formalizing prior work on VQA, called module networks (Andreas, 2016) as discrete, structured, latent variable models on the joint distribution over questions and answers given images, and devise a procedure to train the model effectively. Our results on a dataset of compositional questions about SHAPES (Andreas,  2016)  show that our model generates more interpretable programs and obtains better accuracy on VQA in the low-data regime than prior work.
The paper presents a new approach for performing visual query answering. System responses are programs that can explain the truth value of the answer.
In the paper, both the problems of learning and inference are taken into account.
To answer queries, this system takes as input an image and a question, which is a set of word from a given vocabulary. Then the question is modeled by a plan (a series of operation that must be performed to answer the query).  Finally, the found answer with the plan are returned. To learn the parameters of the model, the examples are tuples composed by an image, a question, the answer, and the program.
Experiments performed on the SHAPES dataset show good performance compared to neural model networks by Johnson et al.
The paper is well written and clear. I have not found any specific problems in the paper, the quality is high and the approach seems to me to be new and worth studying.
The discussion on related work seems to be good, as well as the discussion on the results of the tests conducted.
On page 5, in equation (3) it seems to me that something is missing in J. Moreover, In Algorithm 1, in lines 4 and 9, the B after the arrow should be written in italic.
Overall, there are several typos that must be corrected. I suggest a double check of the English. For example:
- page 3, "as modeling *uncertaintly* should..."
- page 6, "Given this goal, we *consrtuct* a latent *varible* ..."
- page 8, in paragraph "Effect of optimizing the true ELBO", the word "that" is repeated twice in the 3rd row
- page 13, "for the" repeated twice in "Moving average baseline" paragraph. Also, in the last line of this paragraph, the sentence seems incomplete.
Pros
- The results are convincing
- The approach is clearly explained
Cons
- English must be checked