LARGE-SCALE ANSWERER IN QUESTIONER'S MIND FOR VISUAL DIALOG QUESTION GENERATION
Answerer in Questioner's Mind (AQM) is an information-theoretic framework that has been recently proposed for task-oriented dialog systems. AQM benefits from asking a question that would maximize the information gain when it is asked. However, due to its intrinsic nature of explicitly calculating the information gain, AQM has a limitation when the solution space is very large. To address this, we propose AQM+ that can deal with a large-scale problem and ask a question that is more coherent to the current context of the dialog. We evaluate our method on GuessWhich, a challenging task-oriented visual dialog problem, where the number of candidate classes is near 10K. Our experimental results and ablation studies show that AQM+ outperforms the state-of-the-art models by a remarkable margin with a reasonable approximation. In particular, the proposed AQM+ reduces more than 60% of error as the dialog proceeds, while the comparative algorithms diminish the error by less than 6%. Based on our results, we argue that AQM+ is a general task-oriented dialog algorithm that can be applied for non-yes-or-no responses.
=================
Updated Thoughts
=================
I was primarily concerned about a lack of analysis regarding the technical contributions moving from AQM to AQM+. The revisions and author comments here have addressed the specific experiments I've asked for and more generally clarified the contributions made as part of AQM+. I've increased my rating to reflect my increased confidence in this paper. Overall, I think this is a good paper and will be interesting to the community.
I also thank the authors for their substantial efforts to revise the paper and address these concerns.
===========
Strengths:
===========
The approach is a sensible application of AQM to the GuessWhich setting and results in significant improvements over existing approaches both in terms of quantitative results and qualitative examples. 
===========
Concerns:
===========
[A] Technical Novelty is Limited Compared to AQM 
The major departures from the AQM approach claimed in the paper (Section 3.3) are:
	[1] the generation of candidate questions through beam search rather than predefined set 
	[2.1] The approximate answerer being an RNN generating free-form language instead of a binary classifier. 
	[2.2] Dropping the assumption that \tilde p(a_t | c, q_t) = \tilde p (a_t | c, q_t, h_{t-1}). 
	[3] Estimate approximate information gain using subsets of the class and answer space corresponding to the beam-search generated question set and their corresponding answers.
I have some concerns about these:
For [1], the original AQM paper explores this exact setting for GuessWhat in Section 5.2 -- generating the top-100 questions from a pretrained RNN question generator via beam search and ranking them based on information gain. From my understand, this aspect of the approach is not novel.
For [2.1] I disagree that this is a departure from the AQM approach, instead simply an artifact of the experimental setting. The original AQM paper was based in the GuessWhat game in which the answerer could only reply with yes/no/na; however, the method itself is agnostic of this choice. In fact, the detailed algorithm explanation in Appendix A of the AQM paper explicitly discusses the possibility of the answer generator being an RNN model. 
Generally, the modifications to AQM largely seem like necessary, straight-forward adjustments to the problem setting of GuessWhich and not algorithmic advances. That said, the changes make sense and do adapt the method to this more complex setting where it performs quite well!
[B] Design decisions are not well justified experimentally
Given that the proposed changes seem rather minor, it would be good to see strong analysis of their effect. Looking back at the claimed difference from AQM, there appear to be a few ablations missing:
- How useful is generating questions? I would have liked to see a comparison to a Q_fix set samples from training. (This corresponds to difference [1] above.)
- How important is dialog history to the aprxAns model? (This corresponds to difference [2.2] above).
- How important is the choice to restrict to |C| classes? Figure 4b begins to study this question but conflates the experiment by simultaneously increasing |Q| and |A|. (This correspond to difference [3] above.)
[C] No evaluation of Visual Dialog metrics
It would be useful to the community to see if this marked improvement in GuessWhich performance also results in improved ability to predict human response to novel dialogs. I (and I imagine many others) would like to see evaluation on the standard Visual Dialog test metrics. If this introspective inference process improves these metrics, it would significantly strengthen the paper!
[D] No discussion of inference time
It would be useful to include discussion of relative inference time. The AQM framework requires substantially more computation than an non-introspective model. Could authors report this relative increase in inference efficiency (say at K=20)? 
[E] Lack of Comparison to Base AQM
I would expect explicit comparison to AQM for a model named AQM+ or a discussion on why this is not possible.
===========
Minor Things:
===========
- I don't understand the 2nd claimed contribution from the introduction "At every turn, AQM+ generates a question considering the context of the previous dialog, which is desirable in practice." Is this claim because the aprxAns module uses history? 
- Review versions of papers often lack polished writing. I encourage the authors to review their manuscript for future versions with an eye for clarity of terminology, even if it means a departure from established notation in prior work. 
- The RL-QA qualitative results, are these from non-delta or delta? Is there a difference between the two in terms of interpretability? 
===========
Overview:
===========
The modifications made to adapt AQM to the GuessWhich setting presented here as AQM+ seem to be somewhat minor technical contributions. Further, where these difference could be explored in greater detail, there is a lack of analysis. That said, the proposed approach does make significant qualitative and quantitative improvements in the target problem. I'm fairly on the fence for this paper and look forward to seeing additional analysis and the opinions of other reviewers.