We study the potential for interaction in natural language classification. We add a limited form of interaction for intent classification, where users provide an initial query using natural language, and the system asks for additional information using binary or multi-choice questions. At each turn, our system decides between asking the most informative question or making the final classification pre-diction.The simplicity of the model allows for bootstrapping of the system without interaction data, instead relying on simple crowdsourcing tasks. We evaluate our approach on two domains, showing the benefit of interaction and the advantage of learning to balance between asking additional questions and making the final prediction. Responding to natural language queries through simple, single-step classification has been studied extensively in many applications, including user intent prediction (Chen et al., 2019; Qu et al., 2019), and information retrieval (Kang and Kim, 2003; Rose and Levinson, 2004). Typical methods rely on a single user input to produce an output, and do not interact with the user to reduce ambiguity and improve the final prediction. For example, users may under-specify a request due to incomplete understanding of the domain; or the system may fail to correctly interpret the nuances of the input query. In both cases, a low quality decision could be mitigated by further interaction with the user. In this paper we take a low-overhead approach to add limited interaction to intent classification. Our goal is two-fold: (a) study the effect of interaction on the system performance, and (b) avoid the cost and complexities of interactive data collection. We build an interactive system that poses a sequence of binary and multiple choice questions follow-What is the bill length of the bird: shorter, similar, or longer than head? ing the initial user natural language query. Figure 1 illustrates such interactions in two domains, showcasing the opportunity for clarification while avoiding much of the complexity involved in unrestricted natural language interactions. We design our approach not to rely on user interaction during learning, which requires users to handle low quality systems or costly Wizard of Oz experiments. We adopt a Bayesian decomposition of the posterior distributions over intent labels and user responses through the interaction process. We use the posteriors to compute question expected information gain, which allows us to efficiently select the next question at each interaction turn. We bal-1 The images are for illustration only. Our approach does not use images. ance between the potential increase in accuracy and the cost of asking additional questions with a learned policy controller that decides whether to ask additional questions or return the final prediction. We estimate each distribution in our posterior decomposition independently by crowdsourcing initial queries and keywords annotation. We use non-interactive annotation tasks that do not require Wizard-of-Oz style dialog annotations (Kel-ley, 1984; Wen et al., 2017). During training, we train a shared text encoder to compare natural language queries, clarification questions, user answers and classification targets in the same embedding space. This enables us to bootstrap to unseen clarification targets and clarification questions, further alleviating the need of expensive annotation. We evaluate our method on two public tasks: FAQ suggestion (Shah et al., 2018) and bird identification using the text and attribute annotations of the Caltech-UCSD Birds dataset (Wah et al., 2011). The first task represents a virtual assistant application in a trouble-shooting domain, while the second task provides well-defined multiple-choice question annotations and naturally noisy language inputs. We evaluate with both a simulator and human users. Our experiments show that adding user interaction significantly increases the classification accuracy. Given at most five turns of interaction, our approach improves the accuracy of a no-interaction baseline by over 100% on both tasks for simulated evaluation and over 90% for human evaluation. Even a single clarification question provides significant accuracy improvements, 40% for FAQ suggestion and 65% for bird identification in our simulated analysis. Our code and data are available at https://github.com/asappresearch/ interactive-classification . Notation We treat the classification label y , interaction question q and the user response r as random variables. We denote an assignment of a random variable using subscripts, such as y = y i and q = q j . We use superscripts for the observed value of the random variable at a given time step, for example, q t is a question asked at time step t . When clear from the context, we write y i instead of y = y i . For example, p ( r | q j , y i ) denotes the conditional distribution of r given y = y i and q = q j , and p ( r k | q j , y i ) further specifies the corresponding probability when r = r k . An interaction starts with the user providing an initial user query x . At each turn t , the system selects a question q t , to which the user responds with r t , or returns a label y to conclude the interaction. We consider two types of questions: binary and multiple choice questions. The predefined set of possible answers for a question q t is R ( q t ) , where R ( q t ) = { yes , no } for binary questions, or a predefined set of question-specific values for multiple choice questions. We denote an interaction up to time t as X t = ( x, (cid:104) ( q 1 , r 1 ) , . . . , ( q t , r t (cid:105) ) , and the set of possible class labels as Y = { y 1 , . . . , y N } . Figure 1 shows example interactions in our two evaluation domains. Model We model the interactive process using a parameterized distribution over class labels that is conditioned on the observed interaction (Section 4.1), a question selection criterion (Sec-tion 4.2), and a parameterized policy controller (Section 4.5). At each time step t , we compute the belief of each y i  Y conditioned on X t  1 . The trained policy controller decides between two actions: to return the current best possible label or to obtain additional information by asking a question. The model selects the question with the maximal information gain. Given a user response, the model updates the belief over the classification labels. Learning We use crowdsourced data to bootstrap model learning. The crowdsourcing data collection includes two non-interactive tasks. First, we obtain a set of user initial queries X i for each label y i . For example, for an FAQ, How do I sign up for Spring Global Roaming', an annotated potential initial query is Travel out of country'. Second, we ask annotators to assign text tags to each y i , and heuristically convert these tags into a set of question-answer pairs A i = { ( q m , r m ) } M i m =1 , where q m denotes a templated question and r m denotes the answer. For example, the question What is your phone operating system?' can pair with one of the following answers: IOS', Android operating system', Windows operating system' or Not applicable'. We denote this dataset as { ( y i , X i , A i ) } Ni =1 . We describe the data collection process in Section 5. We use this data to train our text embedding model (Section 4.3), to create a user simulator (Section 4.4), and to train the policy controller (Section 4.5). Evaluation We report classification the model accuracy, and study the trade-off between accuracy and the number of turns that the system takes. We evaluate with both a user simulator and real human users. When performing human evaluation, we additionally collect qualitative ratings. Human feedback has been leveraged to train natural language processing models, including for dialogue (Li et al., 2016), semantic parsing (Artzi and Zettlemoyer, 2011; Wang et al., 2016; Iyer et al., 2017) and text classification (Hancock et al., 2018). These methods collect user feedback after the model-predicting stage and treat user feedback as additional offline training data to improve the model. In contrast, our model leverages user interaction to increase prediction performance. Human feedback has been incorporated in reinforcement learning as well, for example to learn a reward function from language as reflecting human preferences (Christiano et al., 2017). Language-based interaction has been studied in the context of visual question answering (de Vries et al., 2017; Lee et al., 2018; Chattopadhyay et al., 2017; Das et al., 2017; Lee et al., 2019; Shukla et al., 2019), SQL generation (Gur et al., 2018; Yao et al., 2019), information retrieval (Chung et al., 2018; Aliannejadi et al., 2019) and multi-turn text-based question answering (Rao and Daume III, 2018; Reddy et al., 2019; Choi et al., 2018). Most methods require learning from recorded dialogues (Wu et al., 2018; Hu et al., 2018; Lee et al., 2018; Rao and Daume III, 2018) or conducting Wizard-of-Oz dialog annotations (Kelley, 1984; Wen et al., 2017). Instead, we limit the interaction to multiple-choice and binary questions. This simplification allows us to reduce the complexity of data annotation while still achieving effective interaction. Our task can be viewed as an instance of the popular 20-question game (20Q), which has been applied to a celebrities knowledge base (Chen et al., 2018; Hu et al., 2018). Our approach differs in using natural language descriptions of classification targets, questions and answers to compute our distributions, instead of treating them as categorical or structural data. Our question selection method is related to several existing methods. Kovashka and Grauman (2013) refine image search by asking to compare visual qualities against selected reference images, and Lee et al. (2018) perform object identification in an image by posing binary questions about the object or its location. Both methods, as well as ours use an entropy reduction criterion to select the best next question. We use a Bayesian decomposition of the joint distribution, which can be easily extended to other model-driven selection methods. Rao and Daume III (2018) propose a learning-to-ask approach by modeling the expected utility of asking question. Our selection method can be considered as a special case when entropy is used as the utility. In contrast to Rao and Daume III (2018), we model the entire interaction history instead of a single turn of follow-up questioning. Our model is trained using crowdsourced annotations, while Rao and Daume III (2018) uses real user-user interaction data. Alternatively to asking questions, Ferecatu and Geman (2007) and Guo et al. (2018) present to the user the most likely image in an image retrieval scenario. The user compares it with the ground-truth image and provides feedback using relevance score or natural language describing the discrepancy between them. We maintain a probability distribution p ( y | X t ) over the set of labels Y . At each interaction step, we first update this belief, decide if to ask a question or return the classification output using a policy controller and, if needed, select a question to ask using information gain. p ( y i | X t ) = p ( y i | X t  1 , q t , r t )  p ( r t , q t , y i | X t  1 ) = p ( q t | y i , X t  1 ) p ( y i | X t  1 ) p ( r t | q t , y i , X t  1 ) . We make two simplifying assumptions as modeling choices. First, the user response depends only on the question q t and the underlying target label y i , and is independent of past interactions. While this independence assumption is unlikely to reflect the course of interactions, it allows to simplify p ( r t | q t , y i , X t  1 ) to p ( r t | q t , y i ) . Second, the selection of the next question q t is deterministic given the interaction history X t  1 . Therefore, p ( q = q t | y i , X t  1 ) = 1 , or zero for q (cid:54) = q t . p ( y i | X t )  p ( r t | q t , y i )  1  p ( y i | X t  1 ) = p ( y i | x ) t (cid:89)  =1 p ( r  | q  , y i ) . (1) Predicting the classification label given the observed interaction X t is reduced to modeling p ( y i | x ) and p ( r k | q j , y i ) , the label y i probability given the initial query x only and the probability of user response r k conditioned on the chosen question q j and class label y i . This factorization enables leveraging separate annotations to learn the two components directly, alleviating the need for collecting costly recordings of user interactions. The system selects the question q t to ask at turn t to maximize the efficiency of the interaction. We use a maximum information gain criterion. Given X t  1 , we compute the information gain on classification label y as the decrease on entropy by observing possible answers to question q : IG ( y ; q | X t  1 ) = H ( y | X t  1 )  H ( y | X t  1 , q ) , where H ( | ) denotes the conditional entropy. Intuitively, the information gain measures the amount of information obtained about the variable y by observing the value of another variable q . Because the first entropy term H ( y | X t  1 ) is a constant regardless of the choice of q , the selection of q t is equivalent to q t = arg min q j H ( y | X t  1 , q j ) , where H ( y | X t  1 , q j ) = (cid:88) r k R ( q j ) p ( r k | X t  1 , q j ) H ( y | X t  1 , q j , r k ) H ( y | X t  1 , q j , r k ) = (cid:88) y i Y p ( y i | X t  1 , q j , r k ) log p ( y i | X t  1 , q j , r k ) p ( r k | X t  1 , q j ) = (cid:88) y i Y p ( r k , y i | X t  1 , q j ) = (cid:88) y i Y p ( r k | q j , y i ) p ( y i | X t  1 ) . We use the independence assumption (Section 4.1) to calculate p ( r k | X t  1 , q j ) . Both p ( r k | X t  1 , q j ) and p ( y i | X t  1 , q j , r k ) can be iteratively updated using p ( y i | x ) and p ( r k | q j , y i ) as the interaction progresses (Equation We model p ( y i | x ) and p ( r k | q j , y i ) by encoding the natural language descriptions of questions, answers and classification labels. In our domains, the text representation of a label is the FAQ document or the bird name. We do not simply treat the labels, questions and answers as categorical variables. Instead, we leverage their natural language content to estimate their correlation This reduces the need for heavy annotation and improves our model in low-resource scenarios. We use a shared neural encoder enc (  ) parameterized by  to encode all texts. Both probability distributions are computed using the dot-product score: S ( u, v ) = enc ( u ) (cid:62) enc ( v ) , where u and v are two pieces of text. The probability of predicting the label y i given an initial query x is: p ( y i | x ) = exp( S ( y i , x )) (cid:80) y j Y exp( S ( y j , x )) . The probability of an answer r k given a question q j and label y i is a linear combination of the observed empirical distribution  p ( r k | q j , y i ) and a parameterized estimation  p ( r k | q j , y i ) : p ( r k | q j , y i ) =   p ( r k | q j , y i )+(1   ) p ( r k | q j , y i ) , where   [0 , 1] is a hyper-parameter. We use the question-answer annotations A i for each label y i to estimate  p ( r k | q j , y i ) using empirical counts. For example, in the FAQ suggestion task, we collect multiple user responses for each question and class label, and average across annotators to estimate  p (Section 5). The second term  p ( r k | q j , y i ) is computed using the text encoder:  p ( r k | q j , y i ) = exp( w  S ( q j # r k , y i ) + b ) (cid:80) r l R ( q j ) exp( w  S ( q j # r l , y i ) + b ) , where w, b  R are scalar parameters and q j # r k is a concatenation of the question q j and the answer r k . 2 Because we do not collect complete annotations to cover every label-question pair,  p provides 2 For example, for a templated question What is your phone operating system?' and an answer IOS', q m = phone operating system' and r m = IOS', therefore, q m # r m = phone operating system IOS'. a smoothing of the partially observed counts using the learned encoding S (  ) . We estimate the parameters  of enc (  ) by pretraining using a dataset { ( y i , X i , A i ) } Ni =1 , where y i is a label, X i is a set of initial queries and A i is a set of question-answer pairs. We create from this data a set of text pairs ( u, v ) to train the scoring function S (  ) . For each label y i , we create pairs ( x, y i ) for each initial query x  X i . We also create ( q m # r m , y i ) for each question-answer pair ( q m , r m )  A i . We minimize the cross-entropy loss using gradient descent: L (  ) =  S ( u, v ) + log (cid:88) v (cid:48) exp( S ( u, v (cid:48) )) . The second term requires summation over all v (cid:48) , which are all the labels in Y . We approximate this sum using negative sampling that replaces the full set Y with a sampled subset in each training batch. The parameters  , w and b are fine-tuned using reinforcement learning during training of the policy controller (Section 4.5). We use a held-out dataset to build a simple simulator. We use the simulator to train the policy controller (Section 4.5) and for performance analysis, in addition to human evaluation. The user simulator provides initial queries to the system and responds to the system initiated clarification questions. The dataset includes N examples { ( y i , X (cid:48) i , A (cid:48) i ) } Ni =1 , where y i is a goal, X (cid:48) i is a set of initial queries and A (cid:48) i = { ( q m , r m ) } M (cid:48) i m =1 is a set of question-answer pairs. While this data is identical in form to our training data, we keep it separated from the data used to estimate S (  ) , p ( y i | x ) and p ( r k | q j , y i ) (Section 4.3). We estimate the simulator question response distribution p (cid:48) ( r k | q j , y i ) using smoothed empirical counts from the data. At the beginning of a simulated interaction, we sample a target label  y , and sample a query x from the associated query set X (cid:48) to start the interaction. Given a system clarification question q t at turn t , the simulator responds with an answer r t  R ( q t ) by sampling from p (cid:48) ( r | q t ,  y ) . Sampling provides natural noise to the interaction, and our model has no knowledge of p (cid:48) . The interaction ends when the system returns a label, which we can then evaluate, for example to compute a reward in Section 4.5. This setup is flexible in that the user simulator can be easily replaced or extended by a real human, and Algorithm 1: Training procedure Estimate p ( y | x ) and p ( r | q, y ) with w and b randomly initialized Estimate p (cid:48) ( r | q, y ) for the user simulator for episode = 1 ...M do Sample ( x,  y ) from dataset for t = 1 ...T do Compute p ( y | X t  1 ) (Equation the system can be further trained with a human-in-the-loop setup. The policy controller decides at each turn t to either select another question to query the user or to conclude the interaction. This provides a tradeoff between exploration by asking questions and exploitation by returning the most probable classification label. The policy controller f (  ,  ;  ) is a feed-forward network parameterized by  that takes the topk probability values and current turn t as input. It generates one of two actions: STOP or ASK . When selecting ASK , a question is selected to maximize the information gain. For STOP , the label y i with highest probability is returned using arg max y i Y p ( y i | X t  1 ) and the interaction ends. Algorithm 1 describes the complete training process. First, we estimate p ( y | x ) and p ( r | q, y ) . We use randomly initialized and fixed w and b parameters. We also estimate p (cid:48) ( r | q, y ) for the user simulator (Section 4.4). We then learn the policy controller using the user simulator with a policy gradient method. We use the REINFORCE algorithm (Williams, 1992). The reward function provides a positive reward for predicting the correct target at the end of the interaction, a negative reward for predicting the wrong target, and a small negative reward for every question asked. We learn the policy controller f (  ,  ;  ) , and estimate w and b in p ( r k | q j , y i ) by back-propagating through the policy gradient. We design a crowdsourcing process to collect data for the FAQ task using Amazon Mechanical Turk. 3 For the Birds domain, we re-purpose an existing dataset. We collect initial queries and tags for each FAQ document. Appendix A.1 describes the worker training process. Initial Query Collection We ask workers to consider the scenario of searching for an FAQ document using an interactive system. Given a target FAQ, we ask for an initial query that they would provide to such a system. The set of initial queries that is collected for each document y i is X i . We encourage workers to provide incomplete information and avoid writing a simple paraphrase of the FAQ. This process provides realistic and diverse utterances because users have limited knowledge of the system and the domain. Tag Collection We collect natural language tag annotations for the FAQ documents. First, we use domain experts to define the set of possible freeform tags. The tags are not restricted to a predefined ontology and can be a phrase or a single word describing the topic of the document. We remove duplicate tags to finalize the set. Experts combine some binary tags to categorical tags. For example, tags IOS', Android operating system' and Windows operating system' are combined to the categorical tag phone operating system'. We use a small set of deterministic, heuristically-designed templates to convert tags into questions. For example, the tag international roaming' is converted into a binary question Is it about international roaming?'; the categorical tag phone operating system' is converted into a multi-choice question What is your phone operating system?'. Finally, we use non-experts to collect user responses to the questions by associating tags with FAQ targets. For binary questions, we ask workers to associate their tags to the FAQ target if they would respond yes' to the question. We show the workers a list of ten tags for a given target as well as a none of the above' option. Annotating all possible target-tag combinations is still expensive and most pairings are negative. We rank the tags based on the relevance against the target using S (  ) trained only 3 https://www.mturk.com/ on the initial queries and show only the current top-50 to the workers. Later, we re-train S (  ) on the complete data. For multi-choice questions, we show the workers a list of possible answers to a tag-generated question for a given FAQ. The workers need to choose one answer that they think best applies. They also have the option of choosing not applicable'. The workers do not engage in a multi-round interactive process. This allows for cheap and scalable collection. Task I: FAQ Suggestion We use the FAQ dataset from Shah et al. (2018). The dataset contains 517 troubleshooting documents from Sprint's technical website. We collect 3,831 initial queries and 118,640 tag annotations using the setup described in Section 5. We split the data into 310/103/104 documents as training, development, and test sets. Only the queries and tag annotations of the 310 training documents are used for pre-training and learning the policy controller, leaving the queries and tag annotations in the development and test splits for evaluation only. Task II: Bird Identification We use the Caltech-UCSD Birds dataset (CUB-200; Wah et al., 2011). The dataset contains 11,788 bird images for 200 different bird species. Each bird image is annotated with a subset of 27 visual attributes and 312 attribute values pertaining to the color or shape of a particular part of the bird. We create categorical questions from attributes with less five possible values, providing eight categorical questions in total. The remaining 279 attributes are converted to binary questions. Each image is annotated with 10 image captions describing the bird in the image (Reed et al., 2016). We use the image captions as initial user queries and bird species as labels. Since each caption contains only partial information about the bird species, the data is naturally noisy and provides challenging user interactions. We do not use the images from the dataset for model training. The images are only provided for grounding during human evaluation. No Interaction: the classification label is predicted using only the initial query. We consider four implementations: (1) BM25: a common keyword-based scoring model for retrieval methods (Robertson and Zaragoza, 2009); (2) RoBERTa BASE : we use a fine-tuned RoBERTa BASE model (Liu et al., 2019) as text encoder; (3) RNN: we use a recurrent neural network (RNN) with simple recurrent unit recurrence (SRU; Lei et al., 2018) as text encoder, together with a fastText word embedding layer (Bojanowski et al., 2017); and (4) RNN + self-attn: the same RNN neural model with a multi-head self-attention layer (Lin et al., 2017; Vaswani et al., 2017). Random Interaction: at each turn, the system randomly selects a question to present the user. After T turns, the classification label is chosen according to the belief p ( y | XT ) . No Initial Query Interaction: the system selects questions without conditioning on the initial user query using maximum information criterion. This is equivalent to using a static decision tree to pick the question, always asking the same first question (Utgoff, 1989; Ling et al., 2004). Variants of Our Approach: we consider several variants of our full model. First, we replace the policy controller with two termination strategies: (1) end the interaction when max p ( y | X t ) passes a threshold, or (2) end the interaction after a fixed number of turns. Second, we disable the parameterized estimator  p ( r k | q j , y i ) by setting  = 1 . Evaluation We use human evaluation, and further analyze performance using our simulator. For human evaluation, users interact with our systems and baseline models using a web-based interactive interface. Each interaction starts with a user scenario: 4 a bird image or a device-troubleshooting scenario described in text. The user types an initial query and answers follow-up questions selected by the system. Once the system returns its prediction, we measure its accuracy, and the user is asked to rate the whole interaction according to rationality and naturalness. 5 The user does not know the correct target label. We use a five-points Likert score for the followup questions. For FAQ Suggestion, we consider two evaluation setups: (1) assuming the model has access to tags in the development and test set for interaction, and (2) using only tags in the 4 Each scenario is related to a single groundtruth label and serves to ground user interactions. 5 We also surveyed users for perceived correctness, but observed it is interpreted identically to rationality. Therefore, we omit this measure. training set annotation. The former is equivalent to adding tags for new documents not seen during training time. The latter zero-shot evaluation setup allows us to investigate the model's performance on unseen targets with no additional tags associated with them. Appendix A.4 provides further details of the human evaluation setup. We do further analysis with the user simulator . We evaluate classification performance using Accuracy@ k , which is the percentage of time the correct target appears among the topk predictions of the model. Implementation Details We use the same encoder to encode initial queries, question-answer pairs and FAQ documents in the FAQ suggestion task. In the bird identification task, where the structure of bird names differs from the other texts, we use one encoder for user initial queries and question-answer pairs and a second encoder for bird names. The policy controller receives a reward of 20 for returning the correct target label, a negative reward of -10 for the wrong target, and a turn penalty of -0.5 for each question asked. For our simulated analysis, we report the averaged results as well as the standard derivation from three independent runs for each model variant and baseline. Appendix A.2 provides more implementation and training details. Our simulated analysis shows that the SRU RNN text encoder performs better or similar to the other encoders. This encoder is also the most lightweight. Therefore, we use it for the majority of our experiments. Human Evaluation Figure 2 and Table 1 show the human evaluation results of our full model and three baselines: our approach with a fixed number of turns (four for FAQ and five for Bird), our approach without access to the initial query (No Init. Query) and our approach without interaction (No Int. (RNN)). Naturalness and rationality measure the quality of the interaction, so we show the results of the user survey in Figure 2 only for interactive systems. Because we do not ask users to fill the end-of-interaction survey for the no interaction baseline, we simply compute its numbers following the first query when evaluating our full approach. Our approach balances between accuracy and the user-centric measures, including naturalness and rationality, achieving stronger performance across Naturalness <latexit sha1_base64="bkWuGBQdkuy27MxXm7MrTV30nsE=">AAACjHicfVFRSxtBEN6cWtOrtbE+9uVsEKQP4U6ECqWQahFfbNWaKCQhzG0mcXFv99idE8Nx0L/Vn9KnvtZ/4SZ3iNXiwLLffvPtzM63cSqFpTD8XfMWFpdeLNdf+q9WXq++aay97VqdGY4drqU2FzFYlEJhhwRJvEgNQhJLPI+v9mf582s0Vmh1RtMUBwlMlBgLDuSoYePHZp/whvKDLyeFX+E9YUb3h31tDHJSaG3hl9Q3oMyALKlKdjqvB1LQtBg2mmErnEfwFEQVaLIqjodrtY3+SPMsQUVcgrW9KExpkIMhwSW6vpnFFPgVTLDnoIIE7SCfT18Em44ZBWNt3FIUzNmHN3JIrJ0msVMmQJf2cW5G/i/Xy2i8O8iFSjNCxctG40wGpIOZlcFIzJyRUweAG+HeGvBLMMDJGe77/a/ohjF45Ap/T9EAafMh74OZJHBT5NX+nEyoUuZ235kaPbbwKehut6KwFZ3sNNvdn6W9dfaOvWdbLGIfWZsdsmPWYZz9Yn/YX3brrXo73ifvcyn1atWXrLN/wju4A5Izy2o=</latexit> <latexit sha1_base64="bkWuGBQdkuy27MxXm7MrTV30nsE=">AAACjHicfVFRSxtBEN6cWtOrtbE+9uVsEKQP4U6ECqWQahFfbNWaKCQhzG0mcXFv99idE8Nx0L/Vn9KnvtZ/4SZ3iNXiwLLffvPtzM63cSqFpTD8XfMWFpdeLNdf+q9WXq++aay97VqdGY4drqU2FzFYlEJhhwRJvEgNQhJLPI+v9mf582s0Vmh1RtMUBwlMlBgLDuSoYePHZp/whvKDLyeFX+E9YUb3h31tDHJSaG3hl9Q3oMyALKlKdjqvB1LQtBg2mmErnEfwFEQVaLIqjodrtY3+SPMsQUVcgrW9KExpkIMhwSW6vpnFFPgVTLDnoIIE7SCfT18Em44ZBWNt3FIUzNmHN3JIrJ0msVMmQJf2cW5G/i/Xy2i8O8iFSjNCxctG40wGpIOZlcFIzJyRUweAG+HeGvBLMMDJGe77/a/ohjF45Ap/T9EAafMh74OZJHBT5NX+nEyoUuZ235kaPbbwKehut6KwFZ3sNNvdn6W9dfaOvWdbLGIfWZsdsmPWYZz9Yn/YX3brrXo73ifvcyn1atWXrLN/wju4A5Izy2o=</latexit> <latexit sha1_base64="bkWuGBQdkuy27MxXm7MrTV30nsE=">AAACjHicfVFRSxtBEN6cWtOrtbE+9uVsEKQP4U6ECqWQahFfbNWaKCQhzG0mcXFv99idE8Nx0L/Vn9KnvtZ/4SZ3iNXiwLLffvPtzM63cSqFpTD8XfMWFpdeLNdf+q9WXq++aay97VqdGY4drqU2FzFYlEJhhwRJvEgNQhJLPI+v9mf582s0Vmh1RtMUBwlMlBgLDuSoYePHZp/whvKDLyeFX+E9YUb3h31tDHJSaG3hl9Q3oMyALKlKdjqvB1LQtBg2mmErnEfwFEQVaLIqjodrtY3+SPMsQUVcgrW9KExpkIMhwSW6vpnFFPgVTLDnoIIE7SCfT18Em44ZBWNt3FIUzNmHN3JIrJ0msVMmQJf2cW5G/i/Xy2i8O8iFSjNCxctG40wGpIOZlcFIzJyRUweAG+HeGvBLMMDJGe77/a/ohjF45Ap/T9EAafMh74OZJHBT5NX+nEyoUuZ235kaPbbwKehut6KwFZ3sNNvdn6W9dfaOvWdbLGIfWZsdsmPWYZz9Yn/YX3brrXo73ifvcyn1atWXrLN/wju4A5Izy2o=</latexit> <latexit sha1_base64="bkWuGBQdkuy27MxXm7MrTV30nsE=">AAACjHicfVFRSxtBEN6cWtOrtbE+9uVsEKQP4U6ECqWQahFfbNWaKCQhzG0mcXFv99idE8Nx0L/Vn9KnvtZ/4SZ3iNXiwLLffvPtzM63cSqFpTD8XfMWFpdeLNdf+q9WXq++aay97VqdGY4drqU2FzFYlEJhhwRJvEgNQhJLPI+v9mf582s0Vmh1RtMUBwlMlBgLDuSoYePHZp/whvKDLyeFX+E9YUb3h31tDHJSaG3hl9Q3oMyALKlKdjqvB1LQtBg2mmErnEfwFEQVaLIqjodrtY3+SPMsQUVcgrW9KExpkIMhwSW6vpnFFPgVTLDnoIIE7SCfT18Em44ZBWNt3FIUzNmHN3JIrJ0msVMmQJf2cW5G/i/Xy2i8O8iFSjNCxctG40wGpIOZlcFIzJyRUweAG+HeGvBLMMDJGe77/a/ohjF45Ap/T9EAafMh74OZJHBT5NX+nEyoUuZ235kaPbbwKehut6KwFZ3sNNvdn6W9dfaOvWdbLGIfWZsdsmPWYZz9Yn/YX3brrXo73ifvcyn1atWXrLN/wju4A5Izy2o=</latexit> Rationality <latexit sha1_base64="FB5LEdO/umPpNVVtJ6aoSx57AYE=">AAACjHicfVFRSxtBEN6c1upZbdTHvlwbBPEh3BVBoRSsivii1WqikIQwt5nExb3dY3dODEfAv+VP6ZOv+i/c5A6xKh1Y9ttvvp3Z+TZOpbAUhn8r3tT0h5mPs3P+/KeFxc/VpeWm1Znh2OBaanMRg0UpFDZIkMSL1CAkscTz+Gp3nD+/RmOFVmc0TLGTwECJvuBAjupWT1fbhDeU7/86Gfkl3hGm93zY1cYgJ4XWPnNHQJkBWXAF9WdSD6Sg4ahbrYX1cBLBWxCVoMbKOO4uVb62e5pnCSriEqxtRWFKnRwMCS7R9cgspsCvYIAtBxUkaDv5ZPpRsOqYXtDXxi1FwYR9eSOHxNphEjtlAnRpX+fG5Hu5Vkb9rU4uVJoRKl406mcyIB2MrQx6YuyMHDoA3Aj31oBfggFOznDfb++hG8bgoSv8O0UDpM163gYzSOBmlJf7/2RCFTK3+87U6LWFb0Hzez0K69HJRm27eVvYO8u+sG9sjUVsk22zA3bMGoyzO3bPHtijt+hteD+8n4XUq5RfssL+CW//CY0/y2o=</latexit> <latexit sha1_base64="FB5LEdO/umPpNVVtJ6aoSx57AYE=">AAACjHicfVFRSxtBEN6c1upZbdTHvlwbBPEh3BVBoRSsivii1WqikIQwt5nExb3dY3dODEfAv+VP6ZOv+i/c5A6xKh1Y9ttvvp3Z+TZOpbAUhn8r3tT0h5mPs3P+/KeFxc/VpeWm1Znh2OBaanMRg0UpFDZIkMSL1CAkscTz+Gp3nD+/RmOFVmc0TLGTwECJvuBAjupWT1fbhDeU7/86Gfkl3hGm93zY1cYgJ4XWPnNHQJkBWXAF9WdSD6Sg4ahbrYX1cBLBWxCVoMbKOO4uVb62e5pnCSriEqxtRWFKnRwMCS7R9cgspsCvYIAtBxUkaDv5ZPpRsOqYXtDXxi1FwYR9eSOHxNphEjtlAnRpX+fG5Hu5Vkb9rU4uVJoRKl406mcyIB2MrQx6YuyMHDoA3Aj31oBfggFOznDfb++hG8bgoSv8O0UDpM163gYzSOBmlJf7/2RCFTK3+87U6LWFb0Hzez0K69HJRm27eVvYO8u+sG9sjUVsk22zA3bMGoyzO3bPHtijt+hteD+8n4XUq5RfssL+CW//CY0/y2o=</latexit> <latexit sha1_base64="FB5LEdO/umPpNVVtJ6aoSx57AYE=">AAACjHicfVFRSxtBEN6c1upZbdTHvlwbBPEh3BVBoRSsivii1WqikIQwt5nExb3dY3dODEfAv+VP6ZOv+i/c5A6xKh1Y9ttvvp3Z+TZOpbAUhn8r3tT0h5mPs3P+/KeFxc/VpeWm1Znh2OBaanMRg0UpFDZIkMSL1CAkscTz+Gp3nD+/RmOFVmc0TLGTwECJvuBAjupWT1fbhDeU7/86Gfkl3hGm93zY1cYgJ4XWPnNHQJkBWXAF9WdSD6Sg4ahbrYX1cBLBWxCVoMbKOO4uVb62e5pnCSriEqxtRWFKnRwMCS7R9cgspsCvYIAtBxUkaDv5ZPpRsOqYXtDXxi1FwYR9eSOHxNphEjtlAnRpX+fG5Hu5Vkb9rU4uVJoRKl406mcyIB2MrQx6YuyMHDoA3Aj31oBfggFOznDfb++hG8bgoSv8O0UDpM163gYzSOBmlJf7/2RCFTK3+87U6LWFb0Hzez0K69HJRm27eVvYO8u+sG9sjUVsk22zA3bMGoyzO3bPHtijt+hteD+8n4XUq5RfssL+CW//CY0/y2o=</latexit> <latexit sha1_base64="FB5LEdO/umPpNVVtJ6aoSx57AYE=">AAACjHicfVFRSxtBEN6c1upZbdTHvlwbBPEh3BVBoRSsivii1WqikIQwt5nExb3dY3dODEfAv+VP6ZOv+i/c5A6xKh1Y9ttvvp3Z+TZOpbAUhn8r3tT0h5mPs3P+/KeFxc/VpeWm1Znh2OBaanMRg0UpFDZIkMSL1CAkscTz+Gp3nD+/RmOFVmc0TLGTwECJvuBAjupWT1fbhDeU7/86Gfkl3hGm93zY1cYgJ4XWPnNHQJkBWXAF9WdSD6Sg4ahbrYX1cBLBWxCVoMbKOO4uVb62e5pnCSriEqxtRWFKnRwMCS7R9cgspsCvYIAtBxUkaDv5ZPpRsOqYXtDXxi1FwYR9eSOHxNphEjtlAnRpX+fG5Hu5Vkb9rU4uVJoRKl406mcyIB2MrQx6YuyMHDoA3Aj31oBfggFOznDfb++hG8bgoSv8O0UDpM163gYzSOBmlJf7/2RCFTK3+87U6LWFb0Hzez0K69HJRm27eVvYO8u+sG9sjUVsk22zA3bMGoyzO3bPHtijt+hteD+8n4XUq5RfssL+CW//CY0/y2o=</latexit> Our Approach <latexit sha1_base64="Uq47Qd6wwkhybOWugrai1+GkJQI=">AAACoXicfVFNaxsxEJW3X+n2y0mPvag1gdKD2Q2F5Jg0pbSHNk6onYBtzKw8tkW0kpBmQ8yy0F/Ye/9Hry2VvUtoktIBMU9vnmakp8wq6SlJfrSiO3fv3X+w8TB+9PjJ02ftza2BN4UT2BdGGXeWgUclNfZJksIz6xDyTOFpdn64qp9eoPPS6K+0tDjOYa7lTAqgQE3ai+0R4SWVHw6Oq7jB76SbXm0OjXMoSKP3V9wXoMKBusadrBuCkrSs4po6Khw/sNYZEItq0u4k3WQd/DZIG9BhTfQmm62Xo6kRRY6ahALvh2liaVyCIykUhiGFRwviHOY4DFBDjn5cri2p+HZgpnxmXFia+Jr9+0QJuffLPAvKHGjhb9ZW5L9qw4Jme+NSalsQalEPmhWKk+Erf/lUruxSywBAOBnuysUCHAgKvxDHo/cYHuPwc2h8ZNEBGfemHIGb53BZlU3+n0zqWhZyHExNb1p4Gwx2umnSTY/fdvYH32p7N9gL9oq9ZinbZfvsI+uxPhPsO/vJfrHfUSf6FPWik1oatZovec6uRTT8A9Y/1Ag=</latexit> <latexit sha1_base64="Uq47Qd6wwkhybOWugrai1+GkJQI=">AAACoXicfVFNaxsxEJW3X+n2y0mPvag1gdKD2Q2F5Jg0pbSHNk6onYBtzKw8tkW0kpBmQ8yy0F/Ye/9Hry2VvUtoktIBMU9vnmakp8wq6SlJfrSiO3fv3X+w8TB+9PjJ02ftza2BN4UT2BdGGXeWgUclNfZJksIz6xDyTOFpdn64qp9eoPPS6K+0tDjOYa7lTAqgQE3ai+0R4SWVHw6Oq7jB76SbXm0OjXMoSKP3V9wXoMKBusadrBuCkrSs4po6Khw/sNYZEItq0u4k3WQd/DZIG9BhTfQmm62Xo6kRRY6ahALvh2liaVyCIykUhiGFRwviHOY4DFBDjn5cri2p+HZgpnxmXFia+Jr9+0QJuffLPAvKHGjhb9ZW5L9qw4Jme+NSalsQalEPmhWKk+Erf/lUruxSywBAOBnuysUCHAgKvxDHo/cYHuPwc2h8ZNEBGfemHIGb53BZlU3+n0zqWhZyHExNb1p4Gwx2umnSTY/fdvYH32p7N9gL9oq9ZinbZfvsI+uxPhPsO/vJfrHfUSf6FPWik1oatZovec6uRTT8A9Y/1Ag=</latexit> <latexit sha1_base64="Uq47Qd6wwkhybOWugrai1+GkJQI=">AAACoXicfVFNaxsxEJW3X+n2y0mPvag1gdKD2Q2F5Jg0pbSHNk6onYBtzKw8tkW0kpBmQ8yy0F/Ye/9Hry2VvUtoktIBMU9vnmakp8wq6SlJfrSiO3fv3X+w8TB+9PjJ02ftza2BN4UT2BdGGXeWgUclNfZJksIz6xDyTOFpdn64qp9eoPPS6K+0tDjOYa7lTAqgQE3ai+0R4SWVHw6Oq7jB76SbXm0OjXMoSKP3V9wXoMKBusadrBuCkrSs4po6Khw/sNYZEItq0u4k3WQd/DZIG9BhTfQmm62Xo6kRRY6ahALvh2liaVyCIykUhiGFRwviHOY4DFBDjn5cri2p+HZgpnxmXFia+Jr9+0QJuffLPAvKHGjhb9ZW5L9qw4Jme+NSalsQalEPmhWKk+Erf/lUruxSywBAOBnuysUCHAgKvxDHo/cYHuPwc2h8ZNEBGfemHIGb53BZlU3+n0zqWhZyHExNb1p4Gwx2umnSTY/fdvYH32p7N9gL9oq9ZinbZfvsI+uxPhPsO/vJfrHfUSf6FPWik1oatZovec6uRTT8A9Y/1Ag=</latexit> <latexit sha1_base64="Uq47Qd6wwkhybOWugrai1+GkJQI=">AAACoXicfVFNaxsxEJW3X+n2y0mPvag1gdKD2Q2F5Jg0pbSHNk6onYBtzKw8tkW0kpBmQ8yy0F/Ye/9Hry2VvUtoktIBMU9vnmakp8wq6SlJfrSiO3fv3X+w8TB+9PjJ02ftza2BN4UT2BdGGXeWgUclNfZJksIz6xDyTOFpdn64qp9eoPPS6K+0tDjOYa7lTAqgQE3ai+0R4SWVHw6Oq7jB76SbXm0OjXMoSKP3V9wXoMKBusadrBuCkrSs4po6Khw/sNYZEItq0u4k3WQd/DZIG9BhTfQmm62Xo6kRRY6ahALvh2liaVyCIykUhiGFRwviHOY4DFBDjn5cri2p+HZgpnxmXFia+Jr9+0QJuffLPAvKHGjhb9ZW5L9qw4Jme+NSalsQalEPmhWKk+Erf/lUruxSywBAOBnuysUCHAgKvxDHo/cYHuPwc2h8ZNEBGfemHIGb53BZlU3+n0zqWhZyHExNb1p4Gwx2umnSTY/fdvYH32p7N9gL9oq9ZinbZfvsI+uxPhPsO/vJfrHfUSf6FPWik1oatZovec6uRTT8A9Y/1Ag=</latexit> Fixed turn <latexit sha1_base64="P82KOR7nKMPHpu3/OQHgy1vAL6c=">AAACtHicfVFbb9MwFHbCZSNc1sIjL4ZqEuKhShAaPG4MTbzANkS7SU1VnTinrTXHjuwT1CqKxN/kcf8Et40qtiGOZJ3P37nZ38lKJR3F8e8gvHf/wcOd3UfR4ydPn+11us+HzlRW4EAYZexlBg6V1DggSQovS4tQZAovsqvjVfziJ1onjf5ByxLHBcy0nEoB5KlJp95PCRdUnxydN1GLP0mbby/HxloUpNG5LfcNqLKgbnDf1w1BSVpuudPK8qOytAbEvInaQXKBOff1upl0enE/Xhu/C5IW9FhrZ5Nu8CrNjagK1CQUODdK4pLGNViSQqEfUTksQVzBDEceaijQjeu1Sg3f90zOp8b6o4mv2b8raiicWxaZzyyA5u52bEX+KzaqaPpxXEtdVoRabAZNK8XJ8JXkPJcrBdXSAxBW+rdyMQcLgvxioij9jP4zFr/6xqclWiBj39Yp2FkBi6Zu/f/SpN6keR95UZPbEt4Fw3f9JO4n5+97h8NfG3l32Uv2mr1hCfvADtkXdsYGTLDrYCfoBN3wIExDEeImNQzalbxgNyzUfwBjWtjq</latexit> <latexit sha1_base64="P82KOR7nKMPHpu3/OQHgy1vAL6c=">AAACtHicfVFbb9MwFHbCZSNc1sIjL4ZqEuKhShAaPG4MTbzANkS7SU1VnTinrTXHjuwT1CqKxN/kcf8Et40qtiGOZJ3P37nZ38lKJR3F8e8gvHf/wcOd3UfR4ydPn+11us+HzlRW4EAYZexlBg6V1DggSQovS4tQZAovsqvjVfziJ1onjf5ByxLHBcy0nEoB5KlJp95PCRdUnxydN1GLP0mbby/HxloUpNG5LfcNqLKgbnDf1w1BSVpuudPK8qOytAbEvInaQXKBOff1upl0enE/Xhu/C5IW9FhrZ5Nu8CrNjagK1CQUODdK4pLGNViSQqEfUTksQVzBDEceaijQjeu1Sg3f90zOp8b6o4mv2b8raiicWxaZzyyA5u52bEX+KzaqaPpxXEtdVoRabAZNK8XJ8JXkPJcrBdXSAxBW+rdyMQcLgvxioij9jP4zFr/6xqclWiBj39Yp2FkBi6Zu/f/SpN6keR95UZPbEt4Fw3f9JO4n5+97h8NfG3l32Uv2mr1hCfvADtkXdsYGTLDrYCfoBN3wIExDEeImNQzalbxgNyzUfwBjWtjq</latexit> <latexit sha1_base64="P82KOR7nKMPHpu3/OQHgy1vAL6c=">AAACtHicfVFbb9MwFHbCZSNc1sIjL4ZqEuKhShAaPG4MTbzANkS7SU1VnTinrTXHjuwT1CqKxN/kcf8Et40qtiGOZJ3P37nZ38lKJR3F8e8gvHf/wcOd3UfR4ydPn+11us+HzlRW4EAYZexlBg6V1DggSQovS4tQZAovsqvjVfziJ1onjf5ByxLHBcy0nEoB5KlJp95PCRdUnxydN1GLP0mbby/HxloUpNG5LfcNqLKgbnDf1w1BSVpuudPK8qOytAbEvInaQXKBOff1upl0enE/Xhu/C5IW9FhrZ5Nu8CrNjagK1CQUODdK4pLGNViSQqEfUTksQVzBDEceaijQjeu1Sg3f90zOp8b6o4mv2b8raiicWxaZzyyA5u52bEX+KzaqaPpxXEtdVoRabAZNK8XJ8JXkPJcrBdXSAxBW+rdyMQcLgvxioij9jP4zFr/6xqclWiBj39Yp2FkBi6Zu/f/SpN6keR95UZPbEt4Fw3f9JO4n5+97h8NfG3l32Uv2mr1hCfvADtkXdsYGTLDrYCfoBN3wIExDEeImNQzalbxgNyzUfwBjWtjq</latexit> <latexit sha1_base64="P82KOR7nKMPHpu3/OQHgy1vAL6c=">AAACtHicfVFbb9MwFHbCZSNc1sIjL4ZqEuKhShAaPG4MTbzANkS7SU1VnTinrTXHjuwT1CqKxN/kcf8Et40qtiGOZJ3P37nZ38lKJR3F8e8gvHf/wcOd3UfR4ydPn+11us+HzlRW4EAYZexlBg6V1DggSQovS4tQZAovsqvjVfziJ1onjf5ByxLHBcy0nEoB5KlJp95PCRdUnxydN1GLP0mbby/HxloUpNG5LfcNqLKgbnDf1w1BSVpuudPK8qOytAbEvInaQXKBOff1upl0enE/Xhu/C5IW9FhrZ5Nu8CrNjagK1CQUODdK4pLGNViSQqEfUTksQVzBDEceaijQjeu1Sg3f90zOp8b6o4mv2b8raiicWxaZzyyA5u52bEX+KzaqaPpxXEtdVoRabAZNK8XJ8JXkPJcrBdXSAxBW+rdyMQcLgvxioij9jP4zFr/6xqclWiBj39Yp2FkBi6Zu/f/SpN6keR95UZPbEt4Fw3f9JO4n5+97h8NfG3l32Uv2mr1hCfvADtkXdsYGTLDrYCfoBN3wIExDEeImNQzalbxgNyzUfwBjWtjq</latexit> No Init. the board. All three models improve the classification performance with the addition of interaction. Qualitatively, the users rate our full approach better than the two other interaction variants. This demonstrates that our model handles effectively real user interaction despite being trained with only non-interactive data. We include additional details in Appendix A.4. Analysis with Simulated Interactions Table 2 shows performance using the the user simulator. We use these results to evaluate different choices beyond what is possible with human studies. We observe interaction is critical; removing the ability to interact decreases performance significantly. The Random Interaction and the No Initial Query Interaction baselines both barely improve the performance over the No Interaction RNN baseline, illustrating the importance of guiding the interaction and considering the initial query. Our full model achieves an Accuracy@1 of 79% for FAQ Suggestion and 49% for Bird Identification using less than five turns, outperforming the No Interaction RNN baseline by 41% and 26%. When having no access to questions and answers in the development and test set during evaluation, the full model performance drops only slightly to 75%, highlighting the model's ability to generalize to unseen tags. The two baselines with alternative termination strategies underperform the full model, indicating the effectiveness of the policy controller. The relatively low performance of the  = 1 variant, which effectively has fewer probability components leveraging natural language than our full model, and No Initial Query Interaction confirm the importance of the learned natural language embedding encoder. Appendix A.3 includes further details on how different text encoders im-pact performance. Figure 3 shows the trade-off between classification accuracy and the number of turns. Each point on the plots is computed by varying the reward turn penalty for our model, the prediction threshold and the predefined number of turns T . Our model with the policy controller or the threshold strategy does not explicitly bound the number of turns, so we report the average number of turns across multiple runs for these two models. We achieve a relative accuracy boost of 40% for FAQ and 65% for Birds over no-interaction baselines with only one clarification question. This highlights the value of leveraging human feedback to improve model accuracy in classification tasks. Figure 4 shows the learning curves of our model with the policy controller trained with different turn penalties r a  { 0 . 5 ,  1 ,  3 } . We observe the models explore during the first 1,000 training episodes in the middle and the right plots. The models achieve relatively stable accuracy after the early exploration stage. The three runs end up using different numbers of expected turns because of the different r a values. We propose an approach for interactive classification, where the system can inquire missing information through a sequence of simple binary or multi-choice questions when users provide under-specified natural language queries. Our expert-guided, incremental design of questions and answers enables easy extension to add new classes, striking the balance between simplicity and extendability. Our modeling choices enable the system to perform zero-shot generalization to unseen classification targets and questions. Our method uses information gain to select the best question to ask FAQ Suggestion Bird Identification Acc@1 Acc@3 Acc@1 Acc@3 No Interaction (BM25) 26% 31% N.A. N.A. No Interaction (RoBERTa BASE ) 30  0 . at every turn, and a lightweight policy to efficiently control the interaction. We demonstrate that the system can be bootstrapped without any interaction data and show effectiveness on two tasks. A potential future research direction is to bridge the gap between this simple bootstrapping paradigm and the incorporation of user free-form responses to allow the system to handle free-text responses. We hope our work will encourage more research on different possibilities of building interactive systems that do not necessarily require handling full-fledged dialogue, but still benefit from user interaction. We thank Derek Chen, Alex Lin, Nicholas Matthews, Jeremy Wohlwend, Yi Yang and the anonymous reviewers for providing valuable feedback on the paper. We would also like to thank Michael Griffths, Anna Folinsky and the ASAPP annotation team for their help on setting up and performing the human evaluation. Finally, we thank Hugh Perkins, Ivan Itzcovich, and Brendan Callahan for their support on the experimental environment setup.
method method abstain result abstain abstain abstain abstain method objective abstain abstain method abstain abstain method method method abstain method abstain abstain method method abstain method abstain method result result result other abstain method method abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain method method method method other other objective other other other method abstain method objective abstain other other objective other method abstain method other other method method other abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain other method abstain method abstain abstain abstain method abstain abstain other method method other method abstain method method method method method abstain method abstain method method abstain abstain method method method abstain method method other abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain method abstain method abstain method method abstain method abstain method abstain abstain method abstain abstain method abstain method abstain abstain abstain other abstain abstain abstain abstain abstain abstain method abstain method method abstain method abstain abstain method abstain abstain method abstain method abstain abstain method abstain abstain abstain abstain method method method method method abstain abstain method abstain method method method method abstain abstain method abstain method method method method abstain method abstain result abstain abstain result abstain result abstain result abstain abstain abstain abstain abstain abstain result result abstain result abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain objective objective method method abstain objective abstain method other other other