Synthesizing QA pairs with a question generator (QG) on the target domain has become a popular approach for domain adaptation of question answering (QA) models. Since synthetic questions are often noisy in practice, existing work adapts scores from a pretrained QA (or QG) model as criteria to select high-quality questions. However, these scores do not directly serve the ultimate goal of improving QA performance on the target domain. In this paper, we introduce a novel idea of training a question value estimator (QVE) that directly estimates the usefulness of synthetic questions for improving the target-domain QA performance. By conducting comprehensive experiments, we show that the synthetic questions selected by QVE can help achieve better target-domain QA performance, in comparison with existing techniques. We additionally show that by using such questions and only around 15% of the human annotations on the target domain, we can achieve comparable performance to the fully-supervised baselines. 1 1 Introduction Question answering (QA) systems based on pretrained language models such as BERT (Devlin et al., 2019) have recently achieved promising performance in machine reading comprehension. However, neural QA systems trained on one domain may not generalize well to another, leaving it challenging to deploy such systems on new domains that lack large-scale QA training data 2 . In this paper, we are interested in semi-supervised domain adaptation : we aim to build a target QA model with source-domain data and a small number of target-domain annotated QA pairs. Due to high annotation costs, existing work (Golub et al., 2017; Dong et al., 2019; Wang et al., 2019; Puri et al., 2020; Chen et al., 2020; Yue et al., 2021) proposes to synthesize target-domain QA pairs via neural question generation (QG) models. The synthetic data are then used to train a QA model on the target domain. In practice, however, the generated questions are often of low quality, such as being semantically mismatched with their paired answers or asking about simple facts (Fig-ure 1). Including all such questions for QA training is less likely to bring substantial improvements. This inspires us to study a crucial problem: 1340 Given a set of target-domain synthetic QA pairs, how to select high-quality ones that are useful to improve target-domain QA training? To address the problem, Alberti et al. (2019) propose the Roundtrip Consistency (RTC) method, which filters 3 questions that cannot be correctly answered by a pretrained QA model. Other work (Shakeri et al., 2020) considers using the generation log likelihood by the QG model (LM Score) as a metric to filter noisy questions (Figure 1, top). Although these filtering techniques have been shown to improve the question quality to some extent (Rennie et al., 2020), they are not directly optimized for selecting questions that can improve QA performance on the target domain . For example, some useful but difficult questions (e.g., the last example in Figure 1) may be filtered by the Roundtrip method, since they cannot be answered correctly by the pretrained QA model. However, these questions are often crucial to further improving QA performance when added into training. In this paper, we propose a question value estimator (QVE) (Figure 1, middle) to select questions that can improve QA performance on the target domain. QVE takes in generated QA examples and outputs real-valued scores (i.e., question values), which are expected to represent the usefulness of generated questions in terms of improving target-domain QA performance. However, training the QVE model towards this goal is challenging due to the lack of supervision (i.e., true question values). To solve the problem, we propose to train the QVE with direct QA feedback from the target domain. Intuitively, if a batch of synthetic questions (when used for training) leads to increasing accuracy of the target-domain QA model, QVE should assign high values to them; the more the accuracy increases, the higher the question values should be. Thus, we optimize QVE with the target-domain QA performance gain after adding the selected questions into training. More formally, given the discrete and non-differentiable question selection process, we formulate the question selection of QVE as a reinforcement learning (Williams, 1992) problem (Figure 2). The QVE receives a batch of synthetic samples each time and learns to select high-quality ones based on their estimated values. The selected samples are then used to train the target-domain QA model, with the resulting performance 3 We interchangeably use filter (noisy/low-quality questions) and select (useful/high-quality questions). gain (on the available target-domain annotations) as the reward. The reward guides the optimization of QVE such that it will eventually make proper question value estimation and selection. To evaluate the QVE model, we instantiate the QG and the QA model based on the pretrained BART (Lewis et al., 2020) and BERT (Devlin et al., 2019), respectively. By carrying out comprehensive experiments on four commonly-used reading comprehension datasets (Trischler et al., 2017; Joshi et al., 2017; Yang et al., 2018; Kwiatkowski et al., 2019), we show that: (1) our QVE model trained with the target-domain QA feedback substantially outperforms the question selection techniques trained without direct QA feedback (Alberti et al., 2019; Shakeri et al., 2020). (2) When using our QVE model to select synthetic questions, QA models can achieve comparable performance to fully-supervised baselines while using only 15% of the full target-domain annotations, which indicates that our method can greatly alleviate human annotation effort in practice. (3) To understand why QVE brings superior improvement, we conduct human evaluation and find that QVE can better identify semantically-matched and difficult questions. Domain Adaptation of Question Answering. In this field, some work (Wiese et al., 2017; Chung et al., 2018; Hazen et al., 2019; Cao et al., 2020) assumes that target-domain annotated questions are available, however, manually creating questions is costly. Therefore, another line of research work (Golub et al., 2017; Wang et al., 2019; Lee et al., 2020; Shakeri et al., 2020) investigates a domain adaptation setting where annotated questions are not available on the target domain. A commonly-adopted approach of this line is to leverage a neural question generation (QG) model (Du et al., 2017; Zhou et al., 2017; Sun et al., 2018; Zhao et al., 2018; Nema et al., 2019; Tuan et al., 2020) to automatically synthesize questions given unlabeled contexts (Du and Cardie, 2018; Zhang and Bansal, 2019; Wang et al., 2019; Liu et al., 2020; Golub et al., 2017; Wang et al., 2019; Lee et al., 2020; Shakeri et al., 2020; Yue et al., 2021); see more discussions in Section 3. However, it is very challenging to achieve satisfying performance without any target annotations. In our work, we study semi-supervised domain adaptation of QA , and assume a small number of target annotations are available , 1341 which can greatly help models adapt to the target domain while requiring minimal human effort. Unsupervised and Semi-supervised QA are two other research topics relevant to our work (Fabbri et al., 2020; Li et al., 2020; Lewis et al., 2019; Dhingra et al., 2018). Unlike domain adaptation, these two settings do not assume the existence of the source domain and synthesize cloze-style questions via rule-based methods for building QA models. Since rule-based QG methods typically have much worse performance than neural ones (pretrained on the source data), we do not compare with these two lines of research in experiments. Data Selection methods aim to select a useful subset from the (noisy) training data. Though (RL-based) data selection methods were explored in other NLP tasks (Ruder and Plank, 2017; Qu et al., 2019; Liu et al., 2019), none of them can be directly applied with trivial efforts to our QA scenario and semi-supervised setting. For example, (Ruder and Plank, 2017) and (Liu et al., 2019) reward or measure the selection with the distribution distance between the selected data and target data, while we reward the selection by measuring how large the improvement the selected data can bring for target-domain QA training, which is more aligned with the end goal. Our work is mostly inspired by recent research on data selection in machine learning community (Ghorbani and Zou, 2019; Jia et al., 2019), particularly (Yoon et al., 2020). However, the significant differences between our work and (Yoon et al., 2020) are as follows: 1) we study a very challenging task, domain adaptation of question answering, which was not studied in (Yoon et al., 2020). How to develop a method in a similar spirit for this task is unexplored. 2) In order to study the task, we begin our method by first proposing two data selection methods that are not covered in (Yoon et al., 2020) but achieve comparable results to existing baselines. We then introduce our RL-based method with a carefully-designed reward, which is well connected to the end goal of improving target-QA performance. the semi-supervised domain adaptation of extractive question answering, where the source-domain and a small number 4 of target-domain QA annotations are provided. Formally, we denote the source-domain QA dataset as D s = { ( c s i , q s i , a s i ) } Ni =1 , where large-scale tuples of context c s i , question q s i , and answer a s i are available. For the target domain, only a small set of annotated QA pairs D t = { ( c t j , q t j , a t j ) } Mj =1 are available ( M (cid:28) N ). Since unlabeled contexts are easy to collect, we assume that they are largely available: C t = { c t l } Ll =1 ( L (cid:29) M ). The task is to build a QA model that can accurately answer questions on the target domain, given D s , D t , and C t . Domain Adaptation via Question Generation. Given the lack of large-scale target-domain annotations, an intuitive approach to domain adaptation is first synthesizing target-domain QA data D t syn = { ( c t l , q t l , a t l ) } Ll =1 automatically from the unlabeled contexts C t , and then training a target-domain QA model on the synthetic ( D t syn ) and the small-size annotated ( D t ) target-domain data. In such an approach, a question generator (QG) g  is first pretrained on the source training data and further finetuned on the available target-domain annotated QA pairs. A well-trained QG model then takes target-domain context-answer pairs as input to generate a question: q t l = g  ( c t l , a t l ) . Although this approach has been shown promising, in practice, its effectiveness is restricted by the quality of synthetic questions. Thus, learning to select ones that can lead to a better target-domain QA model becomes a crucial problem. With respect to how to obtain a t l for QG, in this paper, we assume an answer a t l (i.e., a text span in the context c t l ) is given, following Du et al. (2017). When the answer a t l is not given, it can be extracted from the given context by using an entity recognition tool (Du and Cardie, 2018), a classifier (Puri et al., 2020) or a seq2seq model (Shakeri et al., 2020). Note that noise caused by such answer extraction tools will further lower the overall quality of the synthesized questions. In this paper, we focus on how to select useful synthetic questions in general (i.e., those questions can be synthesized by any QG process) and assume answers are given for simplicity. Given the synthetic target-domain QA data D t syn the task is to select high-quality pairs from D t syn that are useful to improve target-domain QA training. Such a selection decision is often made based on some scores that can indicate the quality of the pairs. For example, Roundtrip filtering (Al-berti et al., 2019) selects questions based on the extracted answer's correctness by a pretrained QA model. Similarly, LM filtering (Shakeri et al., 2020) selects questions with high log-likelihood scores in the generation. However, these scores do not directly serve the goal of improving target-domain QA training. Inspired by recent research on data selection in the machine learning community (Ghor-bani and Zou, 2019; Jia et al., 2019; Yoon et al., 2020), we propose a new idea of training a question value estimator , which predicts the usefulness of a synthetic question for target-domain QA. Formally, we design a question value estimator (QVE), e  , which takes in a synthetic QA example ( c l , q l , a l ) (for simplicity, we omit the superscript t ) and outputs a score indicating its value, i.e., v l = e  ( c l , q l , a l ) . The value can imply the potential for improving the target-domain QA performance when being used as a training sample. With this score, one can select most useful synthetic examples for the target-domain QA training. We use a BERT model as the backbone of the QVE. Specifically, we concatenate the context, question and answer as input to the QVE, and use BERT to encode the sequence (Devlin et al., 2019). h = BERT [ <CLS> q <ANS> a <SEP> c ] where q, a, c represent the question, answer, and context, respectively. h  RH denotes the hidden representation of the input sequence derived from the  <CLS>  token. <ANS> and <SEP> are two special tokens used as delimiters. In our preliminary experiments, we find that adding the answer (start index and end index) probabilities ( p s , p e ) by a pretrained QA model as additional features to the hidden representation h can accelerate the QVE training convergence and lead to better performance. Thus, we add these two features ( p s , p e ) followed by linear transformations of the original hidden representation, and then build a linear classifier to output the question value. h (cid:48) =  ( W 2  ( W 1 h + b 1 ) + b 2 ) h (cid:48)(cid:48) =  ( W 3 ( h (cid:48)  p s  p e ) + b 3 ) v l = W 4 h (cid:48)(cid:48) + b 4 where W 1  RH 1  H , W 2  RH 2  H 1 , W 3  RH 3  H 2 , W 4  RH 3 , b 1  RH 1 , b 2  RH 2 , b 3  RH 3 , b 4  R are trainable parameters of linear layers. is the activation function tanh . Learning such a question value estimator is challenging because we do not have direct supervision on the true value or usefulness of a synthetic question. We discuss two straightforward baselines to train QVE in Section 4.1, and a more advanced one based on reinforcement learning in Section 4.2. Binary Classifier : One straightforward solution is to treat QVE as a binary classifier and train it based on the human-annotated (positive) and the machine-synthesized (negative) QA pairs. Given the scarcity of target-domain data, we first pretrain the classifier on the source domain and then finetune it on the target domain. More specifically, we train a QG model on 70% of the source training data and generate synthetic questions on the remaining 30% of the source training contexts. The generated questions and the source-domain annotated questions are used to train this binary classifier. The classifier is then finetuned based on the small set of target-domain annotations (positive) and the samples synthesized on the same target-domain contexts (negative). However, not all of the generated questions are bad. Simply treating all synthetic samples as negatives may mislead the classifier. Thus, we loose this assumption and introduce a ranking baseline. Ranking Baseline : We assume that the quality of human-annotated questions is not inferior than that of machine-synthesized ones. Thus, we train QVE based on a ranking triplet loss defined as follows: L r = (cid:88) max(0 , m + v s  v h ) where v s , v h are the estimated question values of the machine-synthesized sample and human-annotated sample. m is set to 0 . 15 as the margin. The two baseline methods have two obvious drawbacks: (1) they are trained to differentiate between human-annotated and machine-synthesized samples, which is mismatched with our goal of selecting high-quality samples among machine-synthesized data ; (2) similar as (Alberti et al., 2019; Shakeri et al., 2020), the two baselines are not trained with direct signals that can represent the usefulness of a synthetic question. In the next section, we will introduce a task-specific training 1343 Question Value Estimator Target Annot. Algorithm 1 QVE REINFORCED Training Input : pretrained QA model f  ; target synthetic QA pairs D t syn ; small target annotations D t . Hyperparameters : outer iterations I o , outer batch size B o , inner iterations I n , inner batch size B n , QVE learning rate  o , QA learning rate  n . Output : QVE e  . A well-trained QVE is expected to assign high values to synthetic questions that can improve the target-domain QA performance. Therefore, an intuitive way to measure the value of a synthetic question is to consider the downstream QA performance gain (on the available target annotations) before and after this question is included in the training set. However, this leave-one-out formulation is computationally expensive and time-consuming, given that it can estimate the value of only one single synthetic question in each forward pass. In light of this challenge, we instead estimate question values in a batch-wise fashion. Algorithm 1 and Figure 2 describe the learning process. Generally speaking, we frame the QVE model learning as a reinforcement learning problem (Williams, 1992), and stimulate QVE to assign higher values to more useful questions by using performance-driven rewards. Specially, for a batch of synthetic examples D = { ( c l , q l , a l ) } B o l =1 in the outer training iteration (Line 4-5), the QVE model selects a subset of examples that are most likely to boost the QA performance on the target domain, based on its judgment on their values. Mathematically, the decision-making outcome is represented by the selection vector S = ( s 1 , s 2 , ..., s B o ) , where s l  { 0 , 1 } l = 1 , ..., B o (Line 6-9). The whole batch-level decision making policy   is described as follows: v l = e  ( c l , q l , a l ) s l  Bernoulli ( v l )   ( S|D ) = B o (cid:89) l =1 [ v s l l  (1  v l ) 1  s l ] , where the selection of a certain example ( c l , q l , a l ) is formulated as sampling from a Bernoulli distribution of probability v l (i.e., its estimated question value). We adopt the Bernoulli sampling based on the estimated value v l instead of setting a hard threshold to encourage the policy exploration. The model is rewarded based on how much performance gain the selected examples could bring 1344 when they are used to train the target-domain QA model. To this end, we finetune the QA model f  on the selected batch samples based on L qa , which typically is a cross-entropy loss: L qa =  B o (cid:88) l log P ( a l | q l , c l ;  ) In practice, to stabilize the QVE training, we choose a large outer batch size B o in each outer training iteration. For finetuning the QA model, we pick a relatively smaller inner batch size B n and repeat the training for I n times, such that the QVE-selected samples are fully utilized (Line 10-14). The reward r qve is defined as the QA performance gain on the target-domain annotations D t before ( f  0 ) and after ( f  ) finetuning (Line 15-16), r qve = reward_fn ( f  0 , f  , D t ) where reward_fn is Exact Match (EM) gain 5 . Given the discrete and non-differentiable question selection process, we update the QVE model using the REINFORCE algorithm (Williams, 1992). Mathematically, we aim to minimize: L  =  E S   ( |D ) [ r qve ] . L  =  E S   [ r qve   log   ( S|D )] =  E S   [ r qve   B o (cid:88) l =1 log[ v s l l (1  v l ) 1  s l ]] . (1) Notably, to mitigate the instability in reinforcement learning, we reset the QA model to its pretrained checkpoint at the end of each outer iteration (Line 19), and keep the pretrained QG model unchanged. After training QVE, we can use it to calculate the question value for all the synthetic questions on the target domain. Then we can select top K % synthetic QA pairs as the training corpus to train the target-domain QA model. We use datasets in the MRQA 2019 Shared Task (Fisch et al., 2019), a popular challenge focusing on generalization in reading comprehension. 5 We also tried F1 gain and loss drop as the reward_fn and the EM gain is slightly better than the other two. Specifically, following Shakeri et al. (2020), we use SQuAD 1.1 (Rajpurkar et al., 2016) as the source-domain dataset. For the target-domain datasets, we consider NewsQA (Trischler et al., 2017), Natural Questions (NQ) (Kwiatkowski et al., 2019), HotpotQA (Yang et al., 2018) and TriviaQA (Joshi et al., 2017) as they are commonly used and have sufficient contexts for the QG model to generate synthetic samples. Since there is no test set available for each dataset, we use the original dev set as the test set. Detailed descriptions of each dataset are in Appendix A. For the target-domain datasets, we assume all the contexts and n annotated QA pairs in the original training sets are available for training. We set n = 1000 (about 1%-1.5% of original training sets) as default and discuss the impact of n in Section 6.2. We implement models using the Hugging Face transformers (Wolf et al., 2020) library. We instantiate the QA model with BERT-base-uncased (Devlin et al., 2019), and the QG model with BART-base (Lewis et al., 2020). For training QVE (Algorithm 1), we use BERT-base-uncased model and set H 1 = H 3 = H = 768 and H 2 = 64 for linear layers. To enable a large batch size B o , we use gradient checkpointing (Chen et al., 2016), a technique used for reducing the memory footprint when training deep neural networks. We set I o = 2000 , B o = 80 , I n = 20 , B n = 4 , and  o =  n = 3 e  5 . To select the best QVE checkpoint, we pick the one that achieves the highest reward on the target annotations or the one that leads to the lowest QA training loss. When training (finetuning) QA and QG models (either on source or target domain), we set training epochs as 2 and 3 respectively. Other hyperparameters are set as default in the transformers library. We evaluate the following QA models built on different training data: (1) Source Only Baseline : we train a QA model on the source-domain data. (2) Source + Target Annotations Baseline : we further finetune the (1) Source Only Baseline on the available target annotated QA pairs. (3) QG Baseline (no filtering) : we first pretrain a QG model on the source-domain data and finetune it on the available target annotations. The 1345 Different Filtering Methods Dataset NoFilter RTC LM QVE NewsQA 74,160 33,756 44,485 44,485 NQ 104,071 62,888 62,443 62,443 HotpotQA 72,928 46,273 43,757 43,757 TriviaQA 61,688 26,361 37,013 37,013 Table 1: Number of synthetic examples selected by different methods. QG model is then used to generate synthetic QA samples on the target contexts. We finetune a QA model sequentially on all available data with the order of source  target synthetic  target annotated for all the datasets except TriviaQA 6 . The same QA finetuning strategy will also be used for (4)-(8). (4) RoundTrip Filtering (Alberti et al., 2019): we use the (2) Source + Target Annotation Baseline to extract answers for target synthetic questions and select the ones, whose extracted answers are correct, as the target synthetic training corpus. (5) LM Filtering (Shakeri et al., 2020): we use the log likelihood scores of synthetic questions produced by the QG model in (3) as the filtering criterion. We select top K% samples as the target synthetic training corpus. (8) QVE (RL) : we train QVE based on the direct feedback from target annotations using RL (Sec-tion 4.2), and then use it to select top K% target synthetic samples. (9) Fully-supervised Baseline : we train a QA model on the original target training data. Note that we report the fully-supervised performance here only as the reference and (1)-(8) are not directly comparable to this. The number of the selected synthetic examples of RoundTrip Filtering is determined by the QA model and varies for each dataset. For LM Filtering and QVE, we select top K% (K=60) samples among all synthetic ones and discuss the impact of the synthetic dataset size in Appendix B. We show the statistics of filtered datasets in Table 1. 6 For the TriviaQA dataset, we merge the target synthetic and target annotated dataset into one training file since directly finetuning on the target annotated dataset would hurt the QA performance based on our preliminary experiments. We first discuss the domain adaptation results on the 4 target-domain QA datasets under semi-supervised setting where n = 1 , 000 target-domain QA examples are available. Table 2 shows the overall results of different methods. We summarize key findings as follows: (1) Compared with RoundTrip and LM Filtering, our QVE (RL) achieves the best performance. This is because both baselines are not specifically trained to select useful examples for improving QA performance on the target domain. Our QVE, on the contrary, is trained with a signal that directly reflects the QA performance, which can more accurately estimate the question value and select useful pairs for target-domain QA. (2) Two QVE baselines (binary classifier and ranking baseline) can select some useful questions and achieve comparable performance with RoundTrip and LM Filtering. However, due to the lack of direct QA evaluation feedback, they underperform QVE (RL), which demonstrates the usefulness of the QA feedback during training QVE. In Table 2, we showed that with n ( n =1,000) target annotated QA pairs and the selected high-quality synthetic QA pairs, we can finetune a better QA model on the target domain. In this section, we discuss the influence of n on the target-domain QA performance. The results are shown in Figure 3, and interesting findings include: (1) In general, the performance of all models improves as more target annotations are used. This is intuitive as more annotated pairs can improve both QA and QG training. With a better QG model, the quality of the synthetic questions is improved, which could also lead to better QA models. (2) Our QVE model can often outperform the QG baseline and the filtering baselines. With an optimization objective considering the downstream QA performance, QVE can select more useful questions for improving target-domain QA. (3) The improvement of our QVE compared with baselines is usually larger when more annotated QA pairs are available. This is because our QVE training (with RL) relies on the QA feedback based on the available annotated pairs. With more annotated pairs, the feedback can be more accurate, thus 1346 No. leading to a better QVE for selecting more useful synthetic questions. (4) With 10,000 (around 15% of the original training set) target annotations and the synthetic questions selected by QVE, we can achieve comparable performance with the fully-supervised baseline. This indicates that one can save more annotation budgets when building a target-domain QA model based on our QVE in practice. The results presented in the previous sections are based on BERT-base and BART-base . In this section, we test whether our QVE can still be effective when working with larger models, and select BERT-Large and BART-Large as QA and QG model respectively. When changing the QA (QG) model to its larger alternative, we keep the other one as the base model to better show the difference. We use NaturalQuestions (NQ) and HotpotQA as representative datasets, and show results on them (with 1,000 target annotations). As shown in Table 3, our QVE model can still help improve the performance for larger instantiations of QG/QA. In this section, we aim to gain a better understanding of why QVE helps QA and verify that QVE selects more semantically matched and non-trivial questions, thus benefiting downstream QA. Since automatic metrics cannot often reflect the actual quality of the question selections, we sample 50 generated examples from each target-domain dataset (200 in total), and ask three human annotators to label whether a generated QA pair is semantically matched (i.e., can be selected to train QA) and (if yes) whether it asks about a simple fact. To lower the annotation bias in determining 1347 Question ID in the dataset Context Question Human Labels Selected by models? whether a generated question asks about a simple fact or not, we provide the ground-truth question (the question in the original dataset created by humans) as a reference. If the generated question is simpler than the ground truth, then it would be marked as trivial; otherwise, it is a non-trivial one. Three annotators work independently and we adopt the majority vote for deciding the final labels of a generated QA pair (if disagreement appears). We calculate the precision, recall and F1 between predictions 7 by each filtering method and human labels (for both semantically matched and non-trivial). As shown in Table 5, though three methods obtain a similar precision on all sampled questions, our method has a better recall, especially on the non-trivial questions. This means that our method can select more semantically matched and non-trivial questions, which explains why it leads to better QA performance. We also show some real cases in Figure 1 and Table 4 to further illustrate this point. For example, our QVE selects What was the nickname given to the woman who allegedly provided call girls for prostitution? while the baselines do not pick this semantically matched and non-trivial question. For another example, Who is the founder of CNN , both baselines select it while our QVE filters it out since such a simple question would probably not help further improve QA. We propose a question value estimator to estimate the usefulness of synthetic questions and select useful ones for improving target-domain QA train-7 ing. We optimize QVE with the target-domain QA performance gain after adding the selected questions into training. Our comprehensive experiments demonstrate the superiority of QVE compared with other question selection methods. Additionally, using the synthetic questions selected by QVE and only around 15% of the human annotated data on each target domain, we can achieve comparable performance to the fully-supervised baselines. The authors would thank all the anonymous reviewers and the entire OSU and GMU NLP Group. This research was sponsored in part by NSF IIS-1815674, NSF CAREER #1942980, NSF OAC-2112606, and Ohio Supercomputer Center (OSC, 1987). The views and conclusions contained herein are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S.Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notice herein.
abstain abstain abstain objective result result abstain abstain objective abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain objective abstain abstain objective abstain method method abstain result abstain abstain method result result result other other other other other method abstain other method other abstain abstain abstain abstain method other abstain objective abstain abstain abstain abstain method abstain abstain other abstain abstain abstain abstain abstain abstain abstain method other other other other other other abstain method abstain abstain method method abstain abstain abstain abstain method other abstain abstain method abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain other method abstain method method abstain method abstain abstain method method method method method method method method method method method method method method method method method abstain method method method abstain abstain method abstain method method method method method method abstain method method abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain result result result abstain abstain abstain abstain abstain abstain abstain result abstain result abstain result objective method objective result other other other other