We study learning from user feedback for extractive question answering by simulating feedback using supervised data. We cast the problem as contextual bandit learning, and analyze the characteristics of several learning scenarios with focus on reducing data annotation. We show that systems initially trained on a small number of examples can dramatically improve given feedback from users on model-predicted answers, and that one can use existing datasets to deploy systems in new domains without any annotation, but instead improving the system on-the-fly via user feedback. Explicit feedback from users of NLP systems can be used to continually improve system performance. For example, a user posing a question to a question-answering (QA) system can mark if a predicted phrase is a valid answer given the context from which it was extracted. However, the dominant paradigm in NLP separates model training from deployment, leaving models static following learning and throughout interaction with users. This approach misses opportunities for learning during system usage, which beside several exceptions we discuss in Section 8 is understudied in NLP. In this paper, we study the potential of learning from explicit user feedback for extractive QA through simulation studies. Extractive QA is a popular testbed for language reasoning, with rich prior work on datasets (e.g., Rajpurkar et al., 2016), task design (Yang et al., 2018; Choi et al., 2018), and model architecture development (Seo et al., 2017; Yu et al., 2018). Learning from interaction with users remains relatively understudied, even though QA is well positioned to elicit user feedback. An extracted answer can be clearly visualized within its supporting context, and a language-proficient user can then easily validate Figure 1: Illustration of an interaction setup for learning from user feedback for QA, and its potential. if the answer is supported or not. 1 This allows for simple binary feedback, and creates a contextual bandit learning scenario (Auer et al., 2002; Langford and Zhang, 2007). Figure 1 illustrates this learning signal and its potential. We simulate user feedback using several widely used QA datasets, and use it as a bandit signal for learning. We study the empirical characteristics of the learning process, including its performance, sensitivity to initial system performance, and tradeoffs between online and offline learning. We also simulate zero-annotation domain adaptation, where we deploy a QA system trained from supervised 1 Answers could also come from erroneous or deceitful contexts. This learning scenario can mitigate fundamental problems in extractive QA. It reduces data collection costs, by delegating much of the learning to interaction with users. It can avoid data collection artifacts because the data comes from the actual system deployment, unlike data from an annotation effort that often involves design decisions immaterial to the system's use case. For example, sharing questionand answer-annotator roles (Rajpurkar et al., 2016), which is detrimental to emulate information seeking behavior (Choi et al., 2018). Finally, it gives systems the potential to evolve over time as the world changes (Lazaridou et al., 2021; Zhang and Choi, 2021). Our simulation experiments show that user feedback is an effective signal to continually improve QA systems across multiple benchmarks. For example, an initial system trained with a small amount of SQUAD (Rajpurkar et al., 2016) annotations (64 examples) improves from 18 to 81.6 F1 score, and adapting a SearchQA (Dunn et al., 2017) system to SQUAD through user feedback improves it from 45 to 84 F1 score. Our study shows the impact of initial system performance, trade-offs between online and offline learning, and the impact of source domain on adaptation. These results create the base for future work that goes beyond simulation to use feedback from human users to improve extractive QA systems. Our code is publicly available at https://github.com/ lil-lab/bandit-qa . We study a scenario where a QA model learns from explicit user feedback. We formulate learning as a contextual bandit problem. The input to the learner is a question-context pair, where the context paragraph contains the answer to the question. The output is a single span in the context paragraph that is the answer to the question. Given a question-context pair, the model predicts an answer span. The user then provides feedback about the model's predicted answer, which is used to update the model parameters. We intentionally experiment with simple binary feedback and basic learning algorithms, to provide a baseline for what more advanced methods could achieve with as few assumptions as possible. Background: Contextual Bandit Learning In a stochastic (i.i.d.) contextual bandit learning problem, at each time step t , the learner independently observes a context 2 x ( t )  D sampled from the data distribution D , chooses an action y ( t ) according to a policy  , and observes a reward r ( t )  R . The learner only observes the reward r ( t ) corresponding to the chosen action y ( t ) . The learner aims to minimize the cumulative regret. Intuitively, regret is the deficit suffered by the learner relative to the optimal policy up to a specific time step. Formally, the cumulative regret at time T is computed with respect to the optimal policy    arg max    E ( x,y,r )  ( D, ) [ r ] : RT := T (cid:88) t =1 r  ( t )  T (cid:88) t =1 r ( t ) , (1) where  is the set of all policies, r ( t ) is the reward observed at time t and r  ( t ) is the reward that the optimal policy   would observe. Minimising the cumulative regret is equivalent to maximising the total reward. 3 A key challenge in contextual bandit learning is to balance exploration and exploitation to minimize overall regret. Scenario Formulation Let a question  q be a sequence of m tokens (cid:104) q 1 , . . . , q m (cid:105) and a context paragraph  c be a sequence of n tokens (cid:104) c 1 , . . . , c n (cid:105) . An extractive QA model 4  predicts a span  y = (cid:104) c i , . . . , c j (cid:105) where i, j  [1 , n ] and i  j in the context  c as an answer. When relevant, we denote   as a QA model parameterized by  . We formalize learning as a contextual bandit process: at each time step t , the model is given a question-context pair ( q ( t ) ,  c ( t ) ) , predicts an answer span  y , and receives a reward r ( t )  IR . The learner's goal is to maximize the total reward (cid:80) Tt =1 r ( t ) . This formulation reflects a setup where, given a question-context pair, the QA system interacts with a user, who validates the model-predicted answer in context, and provides feedback which is mapped to numerical reward. 2 The term context here refers to the input to the learner policy, and is different from the term context as we use it later in extractive QA, where the term context refers to the evidence document given as input to the model. 3 Equivalently, the problem is often formulated as loss minimization (Bietti et al., 2018). Learning Algorithm We learn using policy gradient. Our learner is similar to REINFORCE (Sut-ton and Barto, 1998; Williams, 2004), but we use arg max to predict answers instead of Monte Carlo sampling from the model's output distribution. 5 We study online and offline learning, also referred to as onand off-policy. In online learning (Algorithm 1), the model identity is maintained between prediction and update; the parameter values that are updated are the same that were used to generate the output receiving reward. This entails that a reward is only used once, to update the model after observing it. In offline learning (Algorithm 2), this relation between update and prediction does not hold. The learner observes reward, often across many examples, and may use it to update the model many times, even after the parameters drifted arbitrarily far from these that generated the prediction. In practice, we observe reward for the entire length of the simulation ( T steps) and then update for E epochs. The reward is re-weighted to provide an unbiased estimation using inverse propensity score (IPS; Horvitz and Thompson, 1952). We clip the debiasing coefficient to avoid amplifying examples with large coefficients (line 10, Algorithm 2). In general, offline learning is easier to implement because updating the model is not integrated with its deployment. Offline learning also uses a training loop that is similar to optimization practices in supervised learning. This allows to iterate over the data multiple times, albeit with the same feedback signal on each example. However, online learning often has lower regret as the model is updated after each interaction. It may also lead to higher overall performance, because as the model improves early on, it may observe more positive feedback overall, which is generally more informative. We empiri-5 Early experiments showed that sampling is not as bene-ficial as arg max , potentially because of the relatively large output space of extractive QA. Yao et al. (2020) made a similar observation for semantic parsing, and Lawrence et al. (2017) used arg max predictions for bandit learning in statistical machine translation. Table 4 in Appendix A provides our experimental results with sampling. Evaluating Performance We evaluate model performance using token-level F1 on a held-out test set, as commonly done in the QA literature (Ra-jpurkar et al., 2016). We also estimate the learner regret (Equation 1). Computing regret requires access to the an oracle   . We use human annotation as an estimate (Section 3). 6 Comparison to Supervised Learning In supervised learning, the data distribution is not dependent on the model, but on a fixed training set { ( q ( t ) ,  c ( t ) , y ( t ) ) } Tt =1 . In contrast, bandit learners are provided with reward data that depends on the model itself: { ( q ( t ) ,  c ( t ) ,  y ( t ) , r ( t ) ) } Tt =1 where r is the reward for the model prediction  y ( t ) = arg max y   ( y |  q ( t ) ,  c ( t ) ) at time step t . Such feedback can be freely gathered from users interacting with the model, while building supervised datasets requires costly annotation. This learning signal can also reflect changing task properties (e.g., world changes) to allow systems to adapt, and its origin in the deployed system use makes it more robust to biases introduced during annotation. We initialize our model with supervised data, and then simulate bandit feedback using supervised data annotations. Initialization is critical so the model does not return random answers, which are likely to be all bad because of the large output space. We use relatively little supervised data from the same domain for in-domain experiments (Sec-tion 5 and 6) to focus on the data annotation re-6 Our oracle is an estimate because of annotation noise and ambiguity in exact span selection. duction potential of user feedback. For domain adaptation, we assume access to a large amount of training data in the source domain, and no annotated data in the target domain (Section 7). Reward We use supervised data annotations to simulate the reward. If the predicted answer span is an exact match index-wise to the annotated span, the learner observes a positive reward of 1.0, and a negative reward of -0.1 otherwise. 7 This reward signal is stricter than QA evaluation metrics (token-level F1 or exact match after normalization). 8 Noise Simulation We study robustness by simulating noisy feedback via reward perturbation: randomly flipping the binary reward with a fixed probability of 8% or 20% as the noise ratio. 9 4 Experimental Setup Data We use six English QA datasets that provide substantial amount of annotated training data taken from the MRQA training portion (Fisch et al., 2019): SQUAD (Rajpurkar et al., 2016), NewsQA (Trischler et al., 2017), SearchQA (Dunn et al., 2017), TriviaQA (Joshi et al., 2017), HotpotQA (Yang et al., 2018), and NaturalQuestions (NQ; Kwiatkowski et al., 2019). The MRQA benchmark simplifies all datasets so that each example has a single span answer with a limited evidence document length (truncated at 800 tokens). Table 7 in Appendix B provides dataset details. We compute performance measures and learning curves on development sets following prior work (Rajpurkar et al., 2016; Ram et al., 2021). Model We conduct experiments with a pretrained SpanBERT model (Joshi et al., 2020). We fine-tune the pre-trained SpanBERT-base model during initial learning and our simulations. Implementation Details We use Hugging Face Transformers (Wolf et al., 2020). When training initial models with little in-domain supervised data (Section 5; Section 6), we use a learning rate of 3e-5 with a linear schedule, batch size 10, and 10 epochs. We obtain the sets of 64, 256, or 1,024 7 We experimented with other reward values, but did not observe a significant difference in performance (Appendix A). 8 Normalization includes lowercasing, modifying spacing, removing articles and punctuation, etc. NaturalQuestions (NQ; Kwiatkowski et al., 2019) is an exception, with an exact index match measure that has similar strictness. 9 Even without our noise simulation, the simulated feedback inherits the noise from the annotation, either from crowdsourcing or distant supervision. examples from prior work (Ram et al., 2021). 10 For models initially trained on complete datasets (Section 7), we use a learning rate 2e-5 with a linear schedule, batch size 40, and 4 epochs. In simulation experiments, we use batch size 40. We turn off dropout to simulate interaction with users in deployment. For single-pass online learning experiments (Section 5; Section 7), we use a constant learning rate of 1e-5. For offline learning experiments (Section 6), we train the model for 3 epochs on the collected feedback with a linear schedule learning rate of 3e-5. Online experiments with SQUAD, HotpotQA, NQ, and NewsQA take 24h each on one NVIDIA GeForce RTX 2080 Ti; 2.56h for offline. For TriviaQA and SearchQA, each online simulation experiment on one NVIDIA TITAN RTX takes 49.5h; 920h for offline. We simulate a scenario where only a limited amount of supervised data is available, and the model mainly learns from explicit user feedback on predicted answers. We use 64, 256, or 1,024 in-domain annotated examples to train an initial model. This section focuses on online learning, where the learner updates the model parameters after each feedback is observed (Algorithm 1). Figure 2 presents the performance of in-domain simulation with online learning. The performance pattern varies across different datasets. Bandit learning consistently improves performance on SQUAD, HotpotQA, and NQ across different amounts of supervised data used to train the initial model. The performance gain is larger with weaker initial models (i.e., trained on 64 supervised exam-ples): 63.6 on SQUAD, 42.7 on HotpotQA, and 40.0 on NQ. Bandit learning is not always effective on NewsQA, TriviaQA, and SearchQA, especially with weaker initial models. This may be attributed to the quality of training set annotations, which determines the accuracy of reward in our setup. SearchQA and TriviaQA use distant supervision to match questions and relevant contexts from the web, likely decreasing reward quality in our setup. While NewsQA is crowdsourced, Trischler et al. (2017) report relatively low human performance (69.4 F1), possibly indicating data challenges that also decrease our reward quality. Learning progres-10 We use the seed 46 sets publicly available at https: //github.com/oriram/splinter . sion across datasets (Figure 3) shows that initial models trained with 1,024 examples can achieve peak performance with one third or even one quarter of feedback provided. Feedback Noise Simulation Figure 3 shows learning curves with simulated noise via different amounts of feedback perturbation (0%, 8%, or 20%). When perturbation-free simulation is effective, models remain robust to noise: 8% noise results in small fluctuations of the learning curve, but the final performance degrades minimally. Starting with weaker initial models and learning with a higher noise ratio may cause learning to fail (e.g., simulation on SQUAD with 64 initial examples and 20% noise). When online perturbation-free simulation fails, online learning with noisy feedback fails too. Sensitivity Analysis Training Transformer-based models has been shown to have stability issues, especially when training with limited amount of data (Zhang et al., 2021). Our nonstandard training procedure (i.e., one epoch with a fixed learning rate) may further increase instability. We study the stability of the learning process using initial models trained on only 64 in-domain supervised examples on HotpotQA and TriviaQA: the former shows significant performance gain while the latter shows the opposite. We experiment with five initial models trained on different sets of 64 supervised examples, each used to initiate a separate simulation experiment. Four out of five experiments on HotpotQA show performance gains similar to what we observed so far, except one experiment that starts with very low initialization performance. In contrast, nearly all experiments on TriviaQA collapse (mean F1 of 7.3). We also conduct sensitivity analysis with stronger initial models trained with 1,024 examples, and observe that the final performance is stable across runs on both HotpotQA and TriviaQA (standard deviations are 0.5 and 2.6). Table 5 in Appendix B provides detailed performance numbers. We simulate offline bandit learning (Algorithm 2), where feedback is collected all at once with the initial model. The learning scenario follows the previous section: only a limited amount of supervised data is available (64, 256, or 1,024 in-domain examples) to train initial models. Table 1 shows the performance of offline simulation experiments compared to online simulations. We observe mixed results. On SQUAD, HotpotQA, NQ, and NewsQA, offline learning outperforms online learning when using stronger initial models (i.e., models trained on 256 and 1,024 exam-ples). This illustrates the benefit of the more standard training loop, especially with our Transformer-based model that is better optimized with a linear learning rate schedule and multiple epochs, both incompatible with the online setup. On TriviaQA and SearchQA, offline simulation is ineffective regardless of the performance of initial models. This result echoes the learning challenges in the online counterparts on these two datasets. Online vs. Offline Regret Table 2 compares online and offline regret. Regret numbers are averaged over the number of feedback observations. 11 Online learning generally displays lower regret for similar initial models on SQUAD, HotpotQA, and NQ. This is expected because later interactions in the simulation can benefit from early feedback in online learning. In contrast, in our offline scenario, we only update after seeing all examples, so regret numbers depend on the initial model only. Regret results on NewsQA, TriviaQA, and SearchQA are counterintuitive, generally showing that online learning has similar or higher regret. The cases showing significantly higher online regret (64+sim on NewsQA and SearchQA) can be explained by the learning failing, which impacts online regret, but not our offline regret. The others are more complex, and we hypothesize that they may be because of combination of (a) inherent noise in the data; and (b) in cases where online learning is effective, the gap between the strictly-defined reward that is used to compute regret and the relaxed F1 evaluation metric. Further analysis is required for a more conclusive conclusion. Learning from user feedback creates a compelling avenue to deploy systems that target new domains not addressed by existing datasets. The scenario we simulate in this section starts with training a QA model on a complete existing annotated dataset, and deploying it to interact with users and learn from their feedback in a new domain. We do not assume access to any annotated training data in 11 Table 8 in Appendix B lists the percentage of positive feedback in online and offline in-domain simulation. the target domain. We report experiments with online learning. Offline adaptation experiments are discussed in Appendix B.3. Figure 4 shows online domain adaptation performance. On 22/30 configurations, online adaptation introduces significant performance gains ( > 2 F1 score). For example, adapting from TriviaQA and SearchQA to the other four domains improves performance by 2772.8 F1. On HotpotQA, the model initially trained on TriviaQA shows an impressive adaptation, improving from 0.2 F1 to 73 F1. 12 Our simulations show reduced effectiveness when the target domain is either TriviaQA or SearchQA, likely because the simulated feedback is based on noisy distantly supervised data. For SearchQA, the low performance of initial models from other domains may also contribute to the adaptation failure. As expected, this indicates the effectiveness of the process depends on the relation 12 We replicate this result with different model initializations to confirm it is not random. between the source and target domains. SearchQA seems farthest from the other domains, mirroring observations from prior work (Su et al., 2019). Figure 5 shows learning curves for our simulation experiments. Generally, we observe the choice of source and target domains influences adaptation rates. Models quickly adapt to SQUAD, HotpotQA, and NQ, reaching near final performance with a quarter of the total feedback provided. On NewsQA, models initially trained on TriviaQA and SearchQA adapt slower than those initially trained on other three datasets. On TriviaQA, we observe little change in performance throughout simulation. On SearchQA, only the model initially trained on TriviaQA shows a performance gain. Both SearchQA and TriviaQA include context paragraphs from the web, potentially making domain adaptation from one to the other easier. Lastly, we compare bandit learning with initial models trained on a small amount of in-domain data (Section 5) and initial models trained on a large amount of out-of-domain data. Table 3 compares online learning with initial models trained on 1,024 in-domain supervised examples and online domain adaptation with a SQU AD-initialized model. SQUAD initialization provides a robust starting point for all datasets except SearchQA. On four out of five datasets, the final performance is better with SQU AD-initialized model. This is potentially because the model is exposed to different signals from two datasets and overall sees more data, either as supervised examples or through feedback. However, on SearchQA, learning with SQU AD-initialized model performs much worse than learning with the initial model trained on 1,024 in-domain examples, potentially because of the gap in initial model performance (23.5 vs. 65 F1). Bandit learning has been applied to a variety of NLP problems including neural machine translation (NMT; Sokolov et al., 2017; Kreutzer et al., 2018a,b; Mendoncca et al., 2021), structured prediction (Sokolov et al., 2016), semantic parsing (Lawrence and Riezler, 2018), intent recognition (Falke and Lehnen, 2021), and summarization (Gunasekara et al., 2021). Explicit human feedback has been studied as a direct learning signal for NMT (Kreutzer et al., 2018b; Mendoncca et al., 2021), semantic parsing (Artzi and Zettle-moyer, 2011; Lawrence and Riezler, 2018), and summarization (Stiennon et al., 2020). Nguyen et al. (2017) simulates bandit feedback to improve an MT system fully trained on a large annotated dataset, including analyzing robustness to feedback perturbations. Our work shows that simulated bandit feedback is an effective learning signal for extractive question answering tasks. Our work differs in focus on reducing annotation costs by relying on few annotated examples only to train the initial model, or by eliminating the need for in-domain annotation completely by relying on data in other domains to train initial models. Implicit human feedback, where feedback is derived from human behavior rather than explicitly requested, has also been studied, including for dialogue (Jaques et al., 2020) and instruction generation (Kojima et al., 2021). We focus on explicit feedback, but implicit signals also hold promise to improve QA systems. Alternative forms of supervision for QA have been explored in prior work, such as explicitly providing fine-grained information (Dua et al., 2020; Khashabi et al., 2020a). Kratzwald et al. (2020) resembles our setting in seeking binary feedback to replace span annotation, but their goal is to create supervised data more economically. Campos et al. (2020) proposes feedback-weighted learning to improves conversational QA using simulated binary feedback. Their approach relies on multiple samples (i.e., feedback signals) per example, training for multiple epochs online by re-visiting the same questions repeatedly, and tuning two additional hy-perparameters. In contrast, we study improving QA systems via feedback as a bandit learning problem. In both online and offline setups, we assume only one feedback sample per example. We also provide extensive sensitivity studies to the amount of annotations available, different model initialization, and noisy feedback across various datasets. Domain adaptation for QA has been widely studied (Fisch et al., 2019; Khashabi et al., 2020b), including using data augmentation (Yue et al., 2021), adversarial training (Lee et al., 2019), contrastive method (Yue et al., 2021), back-training (Kul-shreshtha et al., 2021), and exploiting small lottery subnetworks (Zhu et al., 2021). We present a simulation study of learning from user feedback for extractive QA. We formulate the problem as contextual bandit learning. We conduct experiments to show the effectiveness of such feedback, the robustness to feedback noise, the impact of initial model performance, the trade-offs between online and offline learning, and the potential for domain adaptation. Our study design emphasizes the potential for reducing annotation costs by annotating few examples or by utilizing existing datasets for new domains. We intentionally adopt a basic setup, including a simple binary reward and vanilla learning algorithms, to illustrate what can be achieved with a relatively simple variant of the contextual bandit learning scenario. Our results already indicate the strong potential of learning from feedback, which more advanced methods are likely to further improve. For example, the balance between online and offline learning can be further explored using proximal policy optimization (PPO; Schulman et al., 2017) or replay memory (Mnih et al., 2015). With well-designed interface, human users may be able to provide more sophisticated feedback (Lamm et al., 2021), which will provide a stronger signal compared to our binary reward. Our aim in this study is to lay the foundation for future work, by formalizing the setup and showing its potential. This is a critical step in enabling future research, especially going beyond simulation to study using real human feedback for QA systems. Another important direction for future work is studying user feedback for QA systems that do both context retrieval and answer generation (Lewis et al., 2020), where assigning the feedback to the appropriate stage in the process poses a challenge. Beyond extractive QA, we hope our work will inspire research of user feedback as a signal to improve other types of NLP systems. Our work's limitations are discussed in Section 1 and Section 9. All six datasets we use are from prior work, are publicly available, and are commonly used for the study of extractive QA. Section 4 reports our computational bud-get and experimental setup in detail. Our code-base is available at https://github.com/ lil-lab/bandit-qa . This research was supported by ARO W911NF-21-1-0106, NSF under grants No. 1750499, the NSF AI Institute for the Foundations of Machine Learning (IFML), and a Google Faculty Research Award. Finally, we thank the action editor and the anonymous reviewers for detailed comments.
method method objective abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain result abstain result abstain other method method abstain abstain abstain abstain result abstain abstain abstain abstain other abstain abstain abstain abstain abstain objective abstain abstain method abstain method method method abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain result method abstain abstain method abstain other abstain abstain method abstain method abstain method method abstain abstain method method abstain abstain method method method method method method abstain abstain method abstain method method method method method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain other other other abstain objective other abstain other abstain other other objective method method other method method result objective result result abstain method objective abstain abstain result abstain method abstain other other other