Interactive Classification by Asking Informative Questions
We propose an interactive classification approach for natural language queries. Instead of classifying given the natural language query only, we ask the user for additional information using a sequence of binary and multiple-choice questions. At each turn, we use a policy controller to decide if to present a question or pro-vide the user the final answer, and select the best question to ask by maximizing the system information gain. Our formulation enables bootstrapping the system without any interaction data, instead relying on non-interactive crowdsourcing an-notation tasks. Our evaluation shows the interaction helps the system increase its accuracy and handle ambiguous queries, while our approach effectively balances the number of questions and the final accuracy.
The authors show that a query classification system can be improved by providing additional information to the system. In particular, additional ‘tags’ are converted to question/answer pairs and the system can choose to ‘ask’ these questions to obtain the ‘answers’ before finalizing its decision. They use a deterministic function (based on information gain) to choose the next question, and train a reinforcement-learned controller to either ask another question or stop. The system is trained without actual interaction, relying on crowdsourced annotation tasks. The authors apply their approach to two different domains: FAQ document suggestion and bird species identification, which seem to be well suited to test their approach. In both cases the authors use a heuristic approach to turn a collection of tags into questions for which annotators then provide answers. The paper is generally well written and organized, and the few minor typos I noticed can be fixed with a spell-checker. I believe the paper is an interesting contribution and I’m leaning towards acceptance. I think that the general question the authors are trying to answer is very interesting: can a system be improved by obtaining additional information from the user. The potential need to disambiguate the user query, and potential underspecification motivate this well. However, the way they set this up the task and created the data is limited though: the system can not dynamically identify an information need and try to fulfill it. It can only add knowledge about pre-defined ‘tags’ to the decision making process. For example, it is not clear to me if a similar improvement could be achieved by identifying a number of relevant tags beforehand and providing these in a single step. So while I am not entirely convinced that the annotation task is a close enough approximation of the real task the authors are trying to solve, I still think it is an interesting result. It might be an interesting additional baseline to train a classifier that has both the query and the relevant tags as input. In addition, ideally I would have liked to see a more thorough analysis of the results. In particular, I would have liked to understand better how the additional information contributes to the classification. I would also like to better understand what value the policy controller provides over choosing a fixed number of questions. For example, how does the performance of the system differ for different numbers of fixed steps? What is the variance in the number of steps the controller chooses? Moreover, the description of the neural model is very short and could be expanded. It would have been interesting to see an analysis of the kind of question the current approach chooses. In addition to that, other heuristics for choosing questions (e.g. increasing diversity) could have been evaluated in addition to information gain. Lastly, the authors say that the code, data and experimental setup will be made public. Are there already concrete plans for doing this and when can it be expected?
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Interactive Classification by Asking Informative Questions
We propose an interactive classification approach for natural language queries. Instead of classifying given the natural language query only, we ask the user for additional information using a sequence of binary and multiple-choice questions. At each turn, we use a policy controller to decide if to present a question or pro-vide the user the final answer, and select the best question to ask by maximizing the system information gain. Our formulation enables bootstrapping the system without any interaction data, instead relying on non-interactive crowdsourcing an-notation tasks. Our evaluation shows the interaction helps the system increase its accuracy and handle ambiguous queries, while our approach effectively balances the number of questions and the final accuracy.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The authors show that a query classification system can be improved by providing additional information to the system. In particular, additional ‘tags’ are converted to question/answer pairs and the system can choose to ‘ask’ these questions to obtain the ‘answers’ before finalizing its decision. They use a deterministic function (based on information gain) to choose the next question, and train a reinforcement-learned controller to either ask another question or stop. The system is trained without actual interaction, relying on crowdsourced annotation tasks. The authors apply their approach to two different domains: FAQ document suggestion and bird species identification, which seem to be well suited to test their approach. In both cases the authors use a heuristic approach to turn a collection of tags into questions for which annotators then provide answers. The paper is generally well written and organized, and the few minor typos I noticed can be fixed with a spell-checker. I believe the paper is an interesting contribution and I’m leaning towards acceptance. I think that the general question the authors are trying to answer is very interesting: can a system be improved by obtaining additional information from the user. The potential need to disambiguate the user query, and potential underspecification motivate this well. However, the way they set this up the task and created the data is limited though: the system can not dynamically identify an information need and try to fulfill it. It can only add knowledge about pre-defined ‘tags’ to the decision making process. For example, it is not clear to me if a similar improvement could be achieved by identifying a number of relevant tags beforehand and providing these in a single step. So while I am not entirely convinced that the annotation task is a close enough approximation of the real task the authors are trying to solve, I still think it is an interesting result. It might be an interesting additional baseline to train a classifier that has both the query and the relevant tags as input. In addition, ideally I would have liked to see a more thorough analysis of the results. In particular, I would have liked to understand better how the additional information contributes to the classification. I would also like to better understand what value the policy controller provides over choosing a fixed number of questions. For example, how does the performance of the system differ for different numbers of fixed steps? What is the variance in the number of steps the controller chooses? Moreover, the description of the neural model is very short and could be expanded. It would have been interesting to see an analysis of the kind of question the current approach chooses. In addition to that, other heuristics for choosing questions (e.g. increasing diversity) could have been evaluated in addition to information gain. Lastly, the authors say that the code, data and experimental setup will be made public. Are there already concrete plans for doing this and when can it be expected?
assistant
The authors show that a query classification system can be improved by providing additional information to the system. In particular, additional ‘tags’ are converted to question/answer pairs and the system can choose to ‘ask’ these questions to obtain the ‘answers’ before finalizing its decision. They use a deterministic function (based on information gain) to choose the next question, and train a reinforcement-learned controller to either ask another question or stop. The system is trained without actual interaction, relying on crowdsourced annotation tasks. The authors apply their approach to two different domains: FAQ document suggestion and bird species identification, which seem to be well suited to test their approach. In both cases the authors use a heuristic approach to turn a collection of tags into questions for which annotators then provide answers. The paper is generally well written and organized, and the few minor typos I noticed can be fixed with a spell-checker. I believe the paper is an interesting contribution and I’m leaning towards acceptance. I think that the general question the authors are trying to answer is very interesting: can a system be improved by obtaining additional information from the user. The potential need to disambiguate the user query, and potential underspecification motivate this well. However, the way they set this up the task and created the data is limited though: the system can not dynamically identify an information need and try to fulfill it. It can only add knowledge about pre-defined ‘tags’ to the decision making process. For example, it is not clear to me if a similar improvement could be achieved by identifying a number of relevant tags beforehand and providing these in a single step. So while I am not entirely convinced that the annotation task is a close enough approximation of the real task the authors are trying to solve, I still think it is an interesting result. It might be an interesting additional baseline to train a classifier that has both the query and the relevant tags as input. In addition, ideally I would have liked to see a more thorough analysis of the results. In particular, I would have liked to understand better how the additional information contributes to the classification. I would also like to better understand what value the policy controller provides over choosing a fixed number of questions. For example, how does the performance of the system differ for different numbers of fixed steps? What is the variance in the number of steps the controller chooses? Moreover, the description of the neural model is very short and could be expanded. It would have been interesting to see an analysis of the kind of question the current approach chooses. In addition to that, other heuristics for choosing questions (e.g. increasing diversity) could have been evaluated in addition to information gain. Lastly, the authors say that the code, data and experimental setup will be made public. Are there already concrete plans for doing this and when can it be expected?