Question Asking as Program Generation
A hallmark of human intelligence is the ability to ask rich, creative, and revealing questions. Here we introduce a cognitive model capable of constructing humanlike questions. Our approach treats questions as formal programs that, when executed on the state of the world, output an answer. The model specifies a probability distribution over a complex, compositional space of programs, favoring concise programs that help the agent learn in the current context. We evaluate our approach by modeling the types of open-ended questions generated by humans who were attempting to learn about an ambiguous situation in a game. We find that our model predicts what questions people will ask, and can creatively produce novel questions that were not present in the training set. In addition, we compare a number of model variants, finding that both question informativeness and complexity are important for producing human-like questions.
The authors examine human question asking where answers are K-ary. They define by hand a PCFG for a “battleship” domain, where there are hidden colored shapes in a partially observable grid (i.e., some tiles are revealed as containing part of a ship of a specific color or being empty). The task of the agent is to ask a question with a single word answer that provides as much information about the state of the board. The PCFG served as a prior over questions, which were defined as statements in lambda calculus. Question goodness was defined as a linear function of its informativeness (expected information gain), its complexity (in terms of its length or negative log probability under of it being generated by the PCFG with a uniform distribution over rewrite rules), and “answer type” (e.g., whether it provides a true/false or color as an answer). The human data came from a previously published paper and was hand coded into their grammar. The authors compared lesioned versions of the model via leave one out cross-validation and found that all factors provided a non-negligible contribution (although complexity was clearly the most important). Finally, they presented a series of novel questions.
Question-asking is an important domain within linguistics and hypothesis testing. The authors provide an innovative perspective unifying modern linguistic and cognitive theory to examine how people ask questions, which has the potential to foster novel research in the active learning literature with models asking more sophisticated questions. The paper is extremely clear and well-written. The model is appropriately evaluated from both quantitative and qualitative perspectives.
From the perspective of a computational cognitive scientist, this paper advances the field of self-directed learning. Previous work found that question informativeness was a poor predictor of human question asking. This followup paper resolves part of why this is the case (it is secondary to complexity and may in part be due to the size of possible questions available to the model). Thus, the paper contributes to the cognitive science literature, which may be sufficient to justify its publication in NIPS.
However, there are serious limitations to their approach, which I can see limiting its interest to the more general NIPS audience. Although it may be possible to automate the human question text to statements in some semantic logic, it is unclear how the approach could scale or extend to domains of interest to the broader community. The battleship domain is small with only a few different features and their method is already computationally strained. That being said, the ideas are interesting enough (active learning with more rich questions) to be of interest, though it may be that the authors will need to provide a clear scenario of interest to the broader community where their technique is likely to be applicable and provide novel insights. 
Minor comments:
1)	It may not be revealing, but I believe you can do a nested model comparison with the LLs of the lesioned models and get tests of statistical significance.
2)	Moving forward, for sequential question asking, there are other models within the linguistics literature of question-asking in formal semantics and pragmatics. A lot of the work comes out of the Questions under Discussion literature. I listed a few references as pointers to this literature.
Ginzburg, J. (1995). Resolving questions, part I. Linguistics and Philosophy, 18(5), 459-527.
Rojas-Esponda, T. (2013). The roadsigns of communication. Proceedings of Semdial (DialDam), 17, 131-139.