Answering compositional questions requiring multi-step reasoning is challenging for current models. We introduce an end-to-end differentiable model for interpreting questions, which is inspired by formal approaches to semantics. Each span of text is represented by a denotation in a knowledge graph, together with a vector that captures ungrounded aspects of meaning. Learned composition modules recursively combine constituents, culminating in a grounding for the complete sentence which is an answer to the question. For example, to interpret not green, the model will represent green as a set of entities, not as a trainable ungrounded vector, and then use this vector to parametrize a composition function to perform a complement operation. For each sentence, we build a parse chart subsuming all possible parses, allowing the model to jointly learn both the composition operators and output structure by gradient descent. We show the model can learn to represent a variety of challenging semantic operators, such as quantifiers, negation, disjunctions and composed relations on a synthetic question answering task. The model also generalizes well to longer sentences than seen in its training data, in contrast to LSTM and RelNet baselines. We will release our code.
This paper proposes for training a question answering model from answers only and a KB by learning latent trees that capture the syntax and learn the semantic of words, including referential terms like "red" and also compositional operators like "not".
I think this model is elegant, beautiful and timely. The authors do a good job of explaining it clearly. I like the modules of composition that seem to make a very intuitive sense for the "algebra" that is required and the parsing algorithm is clean. 
However, I think that the evaluation is lacking, and in some sense the model exposes the weakness of the dataset that it uses for evaluation.
I have 2.5 major issues with the paper and a few minor comments: 
Parsing:
* The authors don't really say what is the base case for \Psi that scores tokens (unless I missed it and if indeed it is missing it really needs to be added) and only provide the recursive case. From that I understand that the only features that they use are whether a certain word makes sense in a certain position of the rule application in the context of the question. While these features are based on Durrett et al.'s neural syntactic parser it seems like a pretty weak signal to learn from. This makes me wonder, how does the parser learn whether one parse is better than the other? Only based on this signal? It makes me suspicious that the distribution of language is not very ambiguous and that as long as you can construct a tree in some context you can do it in almost any other context. This is probably due to the fact that the CLEVR dataset was generated mostly using templates and is not really natural utterances produced by people. Of course many people have published on CLEVR although of its language limitations, but I was a bit surprised that only these features are enough to solve the problem completely, and this makes me curious as to how hard is it to reverse-engineer the way that the language was generated with a context-free mechanism that is similar to how the data was produced.
* Related to that is that the decision for a score of a certain type t for a span (i,j) is the sum for all possible rule applications, rather than a max, which again means that there is no competition between different parse trees that result with the same type of a single span. Can the authors say something about what the parser learns? Does it learn to extract from the noise clear parse trees? What is the distribution of rules in those sums? is there some rule that is more preferred than others usually? It seems like there is loss of information in the sum and it is unclear what is the effect of that in the paper.
Evaluation:
* Related to that is indeed the fact that they use CLEVR only. There  is now the Cornell NLVR dataset that is more challenging from a language perspective and it would be great to have an evaluation there as well. Also the authors only compare to 3 baselines where 2 don't even see the entire KB, so the only "real" baseline is relation net. The authors indeed state that it is state-of-the-art on clevr. 
* It is worth noting that relation net is reported to get 95.5 accuracy while the authors have 89.4. They use a subset so this might be the reason, but I am not sure how they compared to relation net exactly. Did they re-tune parameters once you have the new dataset? This could make a difference in the final accuracy and cause an unfair advantage.
* I would really appreciate more analysis on the trees that one gets. Are sub-trees interpretable? Can one trace the process of composition? This could have been really nice if one could do that. The authors have a figure of a purported tree, but where does this tree come from? From the mode? Form the authors?
Scalability:
* How much of a problem would it be to scale this? Will this work in larger domains? It seems they compute an attention score over every entity and also over a matrix that is squared in the number of entities. So it seems if the number of entities is large that could be very problematic. Once one moves to larger KBs it might become hard to maintain full differentiability which is one of the main selling points of the paper. 
Minor comments:
* I think the phrase "attention" is a bit confusing - I thought of a distribution over entities at first. 
* The feature function is not super clearly written I think - perhaps clarify in text a bit more what it does.
* I did not get how the denotation that is based on a specific rule applycation t_1 + t_2 --> t works. Is it by looking at the grounding that is the result of that rule application?
* Authors say that the neural enquirer and neural symbolic machines produce flat programs - that is not really true, the programs are just a linearized form of a tree, so there is nothing very flat about it in my opinion.
Overall, I really enjoyed reading the paper, but I was left wondering whether the fact that it works so well mostly attests to the way the data was generated and am still wondering how easy it would be to make this work in for more natural language or when the KB is large.