THE NEURO-SYMBOLIC CONCEPT LEARNER: INTERPRETING SCENES, WORDS, AND SENTENCES FROM NATURAL SUPERVISION
We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns visual concepts, words, and semantic parsing of sentences without explicit supervision on any of them; instead, our model learns by simply looking at images and reading paired questions and answers. Our model builds an object-based scene representation and translates sentences into executable, symbolic programs. To bridge the learning of two modules, we use a neuro-symbolic reasoning module that executes these programs on the latent scene representation. Analogical to human concept learning, the perception module learns visual concepts based on the language description of the object being referred to. Meanwhile, the learned visual concepts facilitate learning new words and parsing new sentences. We use curriculum learning to guide the searching over the large compositional space of images and language. Extensive experiments demonstrate the accuracy and efficiency of our model on learning visual concepts, word representations, and semantic parsing of sentences. Further, our method allows easy generalization to new object attributes, compositions, language concepts, scenes and questions, and even new program domains. It also empowers applications including visual question answering and bidirectional image-text retrieval.
To achieve the state-of-the-art on the CLEVR and the variations of this, the authors propose a method to use object-based visual representations and a differentiable quasi-symbolic executor. Since the semantic parser for a question input is not differentiable, they use REINFORCE algorithm and a technique to reduce its variance. 
Quality: 
The issue of invalid evaluation should be addressed. CLEVR dataset has train, validation, and test sets. Since the various hyper-parameters are determined with the validation set, the comparison of state-of-the-art should be done using test set. As the authors mentioned, REINFORCE algorithm may introduce high variance, this notion is critical to report valid results. However, the authors only report on the validation set in Table 2 including the main results, Table 4. For Table 5, they only specify train and test splits. Therefore, I firmly recommend the authors to report on the test set for the fair comparison with the other competitive models, and please describe how to determine the hyperparameters in all experimental settings. 
   
Clarity:
As mentioned above, please specify the experimental details regarding setting hyperparameters.
In Experiments section, the authors used less than 10% of CLEVR training images. How about to use 100% of the training examples? How about to use the same amount of training examples in the competitive models? The report is incomplete to see the differential evident from the efficient usage of training examples.
Originality and significance:
The authors argue that object-based visual representation and symbolic reasoning are the contributions of this work (excluding the recent work, NS-VQA < 1 month). However, bottom-up and top-down attention work [1] shows that attention networks using object-based visual representation significantly improve VQA and image captioning performances. If the object-based visual representation alone is the primary source of improvement, it severely weakens the argument of the neuro-symbolic concept learner. Since, considering the trend of gains, the contribution of the proposing method seems to be incremental, this concern is inevitable. To defend this critic, the additional experiment to see the improvement of the other attentional model (e.g, TbD, MAC) using object-based visual representations, without any other annotations, is needed.
Pros:
- To confirm the effective learning of visual concepts, words, and semantic parsing of sentences, they insightfully exploit the nature of the CLEVR dataset for visual reasoning diagnosis.
Cons:
- Invalid evaluation to report only on the validation set, not test set.
- The unclear significance of the proposed method combining object-based visual representations and symbolic reasoning
- In the original CLEVR dataset paper, the authors said "we stress that accuracy on CLEVR is not an end goal in itself" and "..CLEVR should be used in conjunction with other VQA datasets in order to study the reasoning abilities of general VQA systems." Based on this suggestion, can this work generalize to real-world settings? This paper lacks to discuss its limitation and future direction toward the general problem settings.
Minor comments:
In 4.3, please fix the typos, "born" -> "brown" and "convlutional" -> "convolutional".
[1] Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., & Zhang, L. (2018). Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering. IEEE Computer Vision and Pattern Recognition (CVPR'18).