Answerer in Questioner's Mind: Information Theoretic Approach to Goal-Oriented Visual Dialog
Goal-oriented dialog has been given attention due to its numerous applications in artificial intelligence. Goal-oriented dialogue tasks occur when a questioner asks an action-oriented question and an answerer responds with the intent of letting the questioner know a correct action to take. To ask the adequate question, deep learning and reinforcement learning have been recently applied. However, these approaches struggle to find a competent recurrent neural questioner, owing to the complexity of learning a series of sentences. Motivated by theory of mind, we propose "Answerer in Questioner's Mind" (AQM), a novel information theoretic algorithm for goal-oriented dialog. With AQM, a questioner asks and infers based on an approximated probabilistic model of the answerer. The questioner figures out the answerer's intention via selecting a plausible question by explicitly calculating the information gain of the candidate intentions and possible answers to each question. We test our framework on two goal-oriented visual dialog tasks: "MNIST Counting Dialog" and "GuessWhat?!". In our experiments, AQM outperforms comparative algorithms by a large margin.
Summary: The current work proposes a new framework, Answerer in Questioner’s mind (AQM), for goal-driven visual dialog. Here, questioner has a model of the answer, which it uses to pick a question from a pool of candidate questions, based on information theoretical heuristics. Experiments are shown on (a) MNIST Counting (toy dataset), and (b) GuessWhat?!, showing improvements over competing baselines.  Strengths: (S1) The approach is well motivated by the obverter technique, based on the theory of mind. The paper does a good job of putting forth these arguments.  Weaknesses: (W1) My primary concern is regarding the limited natural language complexity of the Answerer the Questioner is trying to model. From the datasets used and examples shown, Answerer either utters single word answers (classification in case of MNIST Counting) or binary answers (GuessWhat?!). In my opinion, the proposed approach benefits the most when there is richness in language used by Answerer, to which Questioner adopts itself via this internal modeling. (C1) explores another dimension to this.  (W2) The manuscript has a lot of writing errors/typos (few listed at the end). Even though it does not hinder the understanding, it does break the flow of the reader. I urge the authors to carefully proofread their manuscript.  Comment: (C1) Modeling Answerer seems more impactful if Answerer has quirks. For instance, in any application, different human users have different preferences, personas, language styles, etc. Such experiments are missing, which would potentially make this paper stronger. As an example, a color blind Answerer (in case of MNIST counting) should drive the Questioner away from asking color based questions, and so on.  (C2) From my understanding, the model seems to benefit from both YOLO object candidates and the selection of question using proposed approach. On the other side, competing methods seem to be doing this visual inference themselves. Perhaps, include a baseline that also conditions on these proposals.  (C3) Less of a comment and more of a curious question. L141-155 mentions that this approach needs training for a given Answerer. Any thoughts on online training or a low-shot adaptation for a novel answerer?  (C4) L117: Functionality of Q-Sampler has not been introduced so far.  (C5) L122, Eq2: Typo in the equation. Are the bounds \Pi_{j=1}^{t} for the product?  (C6) L169-170: How is 85% -> 46.6%? These two sentences are unclear.  (C7) L212: What does consecutive questions mean here? Aren’t these just pool of questions that make up Q-Sampler, not in any particular sequence?  (C8) L191: Is the class label from YOLO also used?  Typos: L14: Swap closing mark and full stop L44: split -> splits L67: Not sure what the sentence is trying to convey, perhaps a typo somewhere? L102: Typo in the sentence. L115: substituted to -> substituted with / by L115: the equation -> an equation L120: to -> as L137: selects -> select L235: to -> with L235: task training -> task of training