Obtaining training data for multi-hop question answering (QA) is time-consuming and resource-intensive. We explore the possibility to train a well-performed multi-hop QA model without referencing any human-labeled multihop question-answer pairs, i.e. , unsupervised multi-hop QA. We propose MQA-QG, an unsupervised framework that can generate human-like multi-hop training data from both homogeneous and heterogeneous data sources. MQA-QG generates questions by first selecting/generating relevant information from each data source and then integrating the multiple information to form a multi-hop question. Using only generated training data, we can train a competent multi-hop QA which achieves 61% and 83% of the supervised learning performance for the HybridQA and the HotpotQA dataset, respectively. We also show that pretraining the QA system with the generated data would greatly reduce the demand for human-annotated training data. Our codes are publicly available at https: //github.com/teacherpeterpan/Unsupervised-Multi-hop-QA . Extractive Question Answering (EQA) is the task of answering questions by selecting a span from the given context document. Works on EQA can be divided into the single-hop (Rajpurkar et al., 2016, 2018; Kwiatkowski et al., 2019) and multihop cases (Yang et al., 2018; Welbl et al., 2018; Perez et al., 2020). Unlike single-hop QA, which assumes the question can be answered with a single sentence or document, multi-hop QA requires combining disjoint pieces of evidence to answer a question. Though different well-designed neural models (Qiu et al., 2019; Fang et al., 2020) have achieved near-human performance on the multi-hop QA datasets (Welbl et al., 2018; Yang et al., 2018), these approaches rely heavily on the When was the driver ranked in the 4-th position in 2004 United States Grand Prix born? availability of large-scale human annotation. Compared with single-hop QA datasets (Rajpurkar et al., 2016), annotating multi-hop QA datasets is signifi-cantly more costly and time-consuming because a human worker needs to read multiple data sources in order to propose a reasonable question. To address the above problem, we pursue a more realistic setting, i.e. , unsupervised multi-hop QA, in which we assume no human-labeled multi-hop question is available for training, and we explore the possibility of generating human-like multi-hop questionanswer pairs to train the QA model. We study multi-hop QA for both the homogeneous case where relevant evidence is in the textual forms (Yang et al., 2018) and the heterogeneous case where evidence is manifest in both tabular and textual forms (Chen et al., 2020b). Though successful attempts have been made to generate single-hop questionanswer pairs by style transfer (Lewis et al., 2019) or linguistic rules (Li et al., 2020), these methods are not directly applicable to the multi-hop setting as: 1) they cannot integrate information from multiple data sources, and 2) they only handle free-form text but not heterogeneous sources as input contexts. We propose Multi-Hop Question Generator (MQA-QG), a simple yet general framework that decomposes the generation of a multi-hop question into two steps: 1) selecting relevant information from each data source, 2) integrating the multiple information to form a question. Specifically, the model first defines a set of basic operators to retrieve / generate relevant information from each input source or to aggregate different information. Afterwards, we define six reasoning graphs . Each corresponds to one type of multi-hop question and is formulated as a computation graph built upon the operators. We generate multi-hop questionanswer pairs by executing the reasoning graph. Figure 1 shows an example of generating a table-to-text question : a) Given the inputs of (table, text), the F indBridge operator locates a bridge entity that connects the contents between table and text. b) We generate a simple, single-hop question for the bridge entity from the text ( QGwithEnt operator) and generate a sentence describing the bridge entity from the table ( DescribeEnt operator). c) The BridgeBlend operator blends the two generated contents to obtain the multi-hop question. We evaluate our method on two multi-hop QA datasets: HotpotQA (Yang et al., 2018) and HybridQA (Chen et al., 2020b). Questions in HotpotQA reason over multiple texts (homogeneous data), while questions in HybridQA reason over both table and text (heterogeneous data). The experiments show that MQA-QG can generate high-quality multi-hop questions for both datasets. Without using any human-labeled examples, the generated questions alone can be used to train a surprisingly well QA model, reaching 61% and 83% of the F1 score achieved by the fully-supervised setting on the HybridQA and HotpotQA dataset, respectively. We also find that our method can be used in a few-shot learning setting. For example, after pretraining the QA model with our generated data, we can obtain 64 . 6 F1 with only 50 labeled examples in HotpotQA, compared with 21 . 6 F1 without the warm-up training. To the best of our knowledge, this is the first work to investigate unsupervised multi-hop QA. We propose MQA-QG, a novel framework to generate high-quality training data without the need to see any human-annotated multi-hop question. We show that the generated training data can greatly benefit the multi-hop QA system in both unsupervised and few-shot learning settings. Unsupervised Question Answering. To reduce the reliance on expensive data annotation, Unsupervised / Zero-Shot QA has been proposed to train question answering models without any human-labeled training data. Lewis et al. (2019) proposed the first unsupervised QA model which generates synthetic (context, question, answer) triples to train the QA model using unsupervised machine translation. However, the generated questions are unlike human-written questions and tend to have a lot of lexical overlaps with the context. To address this, followup works utilized the Wikipedia cited documents (Li et al., 2020), predefined templates (Fab-bri et al., 2020), or pretrained language model (Puri et al., 2020) to produce more natural questions resembling the human-annotated ones. However, all the existing studies are focused on the SQuAD (Rajpurkar et al., 2016) dataset to answer single-hop and text-only questions. These methods do not generalize to multi-hop QA because they lack integrating and reasoning over disjoint pieces of evidence. Furthermore, they are restricted to text-based QA without considering structured or semi-structured data sources such as KB and Table. In contrast, we propose the first framework for unsupervised multi-hop QA , which can reason over disjoint structured or unstructured data to answer complex questions. Multi-hop Question Generation. Question Generation (QG) aims to automatically generate questions from textual inputs (Pan et al., 2019). Early work of Question Generation (QG) relied on syntax rules or templates to transform a piece of given text to questions (Heilman, 2011; Chali and Hasan, 2012). With the proliferation of deep learning, QG evolved to use supervised neural models, where most systems were trained to generate questions from (passage, answer) pairs in the SQuAD dataset (Du et al., 2017; Zhao et al., 2018; Kim et al., 2019). With the advent of pretraining language models (Dong et al., 2019), the challenge of generating single-hop questions similar to SQuAD Group Operator Inputs  Outputs Description Selection FindBridge (Table T , Text D ) or Texts ( D 1 , D 2 )  Bridge Entities EB Select an entity EB that links the two input texts D 1 and D 2 (or links the table T and the text D ) FindComEnt Text D  Comparative Entities EC Extract potential comparative entities from the input text (location, datetime, number, etc.). have largely been addressed. QG research has started to generate more complex questions that require deep comprehension and multi-hop reasoning (Tuan et al., 2020; Pan et al., 2020; Xie et al., 2020; Yu et al., 2020). For example, Tuan et al. (2020) proposed a multi-state attention mechanism to mimic the multi-hop reasoning process. Pan et al. (2020) parsed the input passage as a semantic graph to facilitate the reasoning over different entities. However, these supervised methods require large amounts of human-written multi-hop questions as training data. Instead, we propose the first unsupervised QG system to generate multi-hop questions without the need to access those annotated data. The setup of Multi-hop QA is as follows. Given a question q and a set of input contexts C = { C 1 ,    , C n } , where each context C i can be a passage, table, image, etc., the QA model p  ( a | q, C ) predicts the answer a for the question q by integrating and reasoning over information from C . In this paper, we consider two-hop questions and denote the required contexts as C i and C j . Formally, each time our model takes as inputs (cid:104) C i , C j (cid:105) to generate a set of ( q, a ) pairs. We focus on two modalities: the heterogeneous case where C i , C j are table and text and the homogeneous case where C i , C j are both texts. However, the design of our framework is flexible enough to generalize to multihop QA for other modalities. components: operators , reasoning graphs , and question filtration . Operators are atomic operations implemented by rules or off-the-shelf pretrained models to retrieve, generate, or fuse relevant information from input contexts ( C i , C j ) . Different reasoning graphs define different types of reasoning chains for multi-hop QA with the operators as building blocks. Training ( q, a ) pairs are generated by executing the reasoning graphs. Question filtration removes irrelevant and unnatural ( q, a ) pairs to give the final training set D for multi-hop QA. In Table 1, we define eight basic operators and divide them into three types: 1) selection : retrieve relevant information from a single context, 2) generation : generate information from a single context, and 3) fusion : fuse multiple retrieved/generated information to construct multi-hop questions. FindBridge : Most multi-hop questions rely on the entities that connect different input contexts, i.e. , bridge entities , to integrate multiple pieces of information (Xiong et al., 2019). FindBridge takes two contexts ( C i , C j ) as inputs, and extracts the entities that appear in both C i and C j as bridge entities. For example, in Figure 1, we extract Jenson Button as the bridge entity. FindComEnt : When generating comparative-type multi-hop questions, we need to decide what property to compare for the bridge entity. FindComEnt extracts potential comparative properties Medal Championship Name Event Silver 2010 Pruszkow Tim Veldt Men's omnium Bronze 2011 Apeldoorn Kristen Wild Women's omnium Gold 2013 Apeldoorn Elis Ligtlee Women's keirin Gold 2013 Apeldoorn Elis Ligtlee Women's sprint Input Table + Target Entity Netherlands at the European Track Championships The table title is Netherlands at the European Track Championships . from the input text. We extract entities with NER types Nationality , Location , DateT ime , and Number from the input text as comparative properties ( cf , Comparison in Figure 4). QGwithAns , QGwithEnt : These two operators generate simple, single-hop questions from a single context, which are subsequently used to compose multi-hop questions. We use the pretrained Google T5 model (Raffel et al., 2019) fine-tuned on SQuAD to implement these two operators. Given the SQuAD training set of context-question-answer triples D = { ( c, q, a ) } , we jointly fine-tune the model on two tasks. 1) QGwithAns aims to generate a question q with a as the answer, given ( c, a ) as inputs. 2) QGwithEnt aims to generate a question q that contains a specific entity e , given ( c, e ) as inputs. The evaluation of this T5-based model can be found in Appendix A.1. DescribeEnt : Given a table T and a target entity e in the table, the DescribeEnt operator generates a sentence that describes the entity e based on the information in the table T . We implement this using the GPT-TabGen model (Chen et al., 2020a) shown in Figure 2. The model first uses template to flatten the table T into a document PT and then feed PT to the pre-trained GPT-2 model (Radford et al., 2019) to generate the output sentence Y . To avoid irrelevant information in PT , we apply a template that only describes the row where the target entity locates. We then finetune the model on the ToTTo dataset (Parikh et al., 2020), a large-scale dataset of controlled table-to-text generation, by maximizing the likelihood of p ( Y | PT ;  ) , with  denoting the model parameters. The implementation details and the model evaluation are in Appendix A.1. QuesToSent : This operator convert a question q into its declarative form s by applying the linguistic rules defined in Demszky et al. (2018). : What is the birthdate of Kirsten Wild? Answer : 15 October 1982  : Kirsten Wild of Netherlands won the bronze medal in the 2011 Apeldoorn. What is the birthdate of the _____ that of Netherlands won the bronze medal in the 2011 Apeldoorn? What is the birthdate of the athlete that of Netherlands won the bronze medal in the 2011 Apeldoorn? Answer : 15 October 1982 Figure 3: An example of the BridgeBlend operator. BridgeBlend : The operator composes a bridge-type multi-hop question based on: 1) a bridge entity e , 2) a single-hop question q that contains e , and 3) a sentence s that describes e . As exemplified in Figure 3, we implement this by applying a simple yet effective rule that replaces the bridge entity e in q with the [MASK] that s  and employ the pretrained BERT-Large (Devlin et al., 2019) to fill in the [MASK] word. CompBlend : This operator composes a comparison-type multi-hop question based on two single-hop questions q 1 and q 2 . The two questions ask about the same comparative property p for two different entities e 1 and e 2 . We form the multi-hop question by filling p , e 1 , and e 2 into pre-defined templates (Further details in Appendix A.2). Based on the basic operators, we define six types of reasoning graphs to generate questions with different types. Each reasoning graph is represented as a directed acyclic graph (DAG) G , where each node in G corresponds to an operator. A node s i is connected by an incoming edge (cid:104) s j , s i (cid:105) if the output of s j is given as an input to s i . As shown in Figure 4, Table-Only and Text-Only represent single-hop questions from table and text, respectively. The remaining reasoning graphs define four types of multi-hop questions. 1) Table-to-Text : bridge-type question between table and text, where the answer comes from the text. 2) Text-to-Table : bridge-type question between table and text, where the answer comes from the table. 3) Text-to-Text : bridge-type question between two texts. 4) Comparison : comparison-type question based on two passages. These four reasoning chains can cover a large portion of questions in existing multihop QA datasets, such as HotpotQA and HybridQA. We generate QA pairs by executing each reasoning graph. Our framework can easily extend to other modalities and reasoning chains by defining new operators and reasoning graphs. Finally, we employ two methods to refine the quality of generated QA pairs. 1) Filtration . We use a pretrained GPT-2 model to filter out those questions that are disfluent or unnatural. The top N samples with the lowest perplexity scores are selected as the generated dataset to train the multi-hop QA model. 2) Paraphrasing . We train a question paraphrasing model based on the BART model (Lewis et al., 2020) to paraphrase each generated question. Our experiments show that filtration brings noticeable improvements to the QA model. However, we show in Section 4.5 that paraphrasing produces more human-like questions but introduces the semantic drift problem that harms the QA performance. We evaluate our framework on two multi-hop QA datasets: HotpotQA (Yang et al., 2018) and HybridQA (Chen et al., 2020b). HotpotQA focuses on multi-hop QA over homogeneous inputs, while HybridQA deals with multi-hop QA over heterogeneous information. HotpotQA contains  100K crowd-sourced multi-hop questions, where each question requires reasoning over two supporting Wikipedia documents to infer the answer. HybridQA contains  70K human-labeled multi-hop questions, where each question is aligned with a structured Wikipedia table and multiple passages linked with the entities in the table. The questions Split Train Dev Test Total HotpotQA Bridge 72,991 5,918  78,909 (81 %) Comparison 17,456 1,487  18,943 (19 %) Total 90,447 7,405  97,852 HybridQA In-Passage 35,215 2,025 2,045 39,285 (56 %) In-Table 26,803 1,349 1,346 29,498 (43 %) Compute 664 92 72 828 (1.1 %) Total 62,682 3,466 3,463 69,611 Table 2: Basic statistics of HotpotQA and HybridQA. are designed to aggregate both tabular information and text information, i.e. , lack of either form renders the question unanswerable. Table 2 shows the statistics of these two datasets and Appendix B.1 gives their data examples. There are two types of multi-hop questions in HotpotQA: bridge-type (81%) and comparison-type (19%). For HybridQA, questions are divided by whether their answers come from the table (In-Table question, 56%) or from the passage (In-Passage question, 44%). Around 80% HybridQA questions requires bridge-type reasoning. Question Generation. In HybridQA, we extract its tabletext corpus consisting of ( T, D ) input pairs, where T denotes the table and set of its linked passages D . We generate two multi-hop QA datasets Q tbl  txt and Q txt  tbl with MQA-QG by executing the Table-to-Text and Text-to-Table reasoning graphs for each ( T, D ) , resulting in a total of 170K QA pairs. We then apply question filtration to obtain the training set Q hybrid with 100K QA pairs. Similarly, for HotpotQA, we first generate Q bge and Q com , which contains only the bridge-type questions and only the comparison-type questions, respectively. Afterward, we merge them and filter the questions to obtain the final training set Q hotpot with 100K QA pairs. In Appendix B.2, we gives the statistics of all the generated datasets. Question Answering For HybridQA, we use the HYBRIDER (Chen et al., 2020b) as the QA model, which breaks the QA into linking and reasoning to cope with heterogeneous information, achieving the best result in HybridQA. For HotpotQA, we use the SpanBERT (Joshi et al., 2020) since it achieved promising results on HotpotQA with reproducible codes. We use the standard Exact Match (EM) and F 1 metrics to measure the QA performance. Baselines. We compare MQA-QG with both supervised and unsupervised baselines. For HybridQA, we first include the two supervised baselines Table-Only and Passage-Only in Chen et al. (2020b), which only rely on the tabular information or the textual information to find the answer. As we are the first to target unsupervised QA on HybridQA, there is no existing unsupervised baseline for direct comparison. Therefore, we construct a strong baseline QDMR-to-Question that generate questions from Question Decomposition Meaning Representation (QDMR) (Wolfson et al., 2020), a logical representation specially designed for multihop questions. We first generate QDMR expressions from the input (table, text) using pre-defined templates and then train a Seq2Seq model (Bah-danau et al., 2014) to translate QDMR into question. Details of this baseline are introduced in Appendix C. For HotpotQA, we introduce three unsupervised baselines. SQuAD-Transfer trains SpanBERT on SQuAD and then transfers it for multi-hop QA. Bridge-Only / Comparison-Only use only the bridge-type / comparison-type questions by MQA-QG to train the QA model. Performance Comparison. Table 3 and Table 4 summarizes the QA performance on HybridQA and HotpotQA, respectively. For HybridQA, we use the reported performance of HYBRIDER as the supervised benchmark (S3) and apply the same model setting of HYBRIDER to train the unsupervised version, i.e. , using our generated QA pairs as the training data (U2 and U3). For HotpotQA, the original paper of SpanBERT only reported the results for the MRQA-2019 shared task (Fisch et al., 2019), which only includes the bridge-type questions in HotpotQA. Therefore, we retrain the SpanBERT on the full HotpotQA dataset to get the supervised benchmark (S4) and using the same model setting to train the unsupervised versions (U7 and U8). Our unsupervised model MQA-QG attains 30.5 F 1 on the HybridQA test set and 68.6 F 1 on the HotpotQA dev set, outperforming all the unsupervised baselines (U1, U4, U5, U6) by large margins. Without using their human-annotated training data, the F 1 gap to the fully-supervised version is only 19 . 5 and 14 . 2 for HybridQA and HotpotQA, respectively. In particular, the results of U2 and U3 even outperform the two weak supervised baselines (S1 and S2) in HybridQA. This demonstrates the effectiveness of MQA-QG in generating good multi-hop questions for training the QA model. To understand the impact of different components in MQA-QG, we perform an ablation study on the HybridQA development set. In Table 5, we compare our full model (A7) with six ablation settings by removing certain the model components (A1 A4) or by restricting the reasoning types (A5 and A6). We make three key observations. Single-hop questions vs. multi-hop questions. A1 to A3 generates single-hop questions using the reasoning graph of Text-Only (A1), Table-Only (A2), or a union of them (A3). Afterwards, we use them to train the HYBRIDER model and test the multi-hop QA performance. In these cases, the model is trained to answer questions based on either table or text but lacking the ability to reason between table and text. As shown in Table 5, A1A3 achieves a low performance of EM and F1, especially for In-Passage questions, showing that single-hop questions alone are insufficient to train a good multi-hop QA system. This reveals that learning to reason between different contexts is essential for multi-hop QA and justifies the necessity of generating multi-hop questions. However, for HotpotQA, we observe that the benefit of multihop questions is not as evident as in HybridQA: the SQuAD-Transfer (U6) achieves a relatively good F1 of 62 . 8 . A potential reason is that the examples of HotpotQA contain reasoning shortcuts through which models can directly locate the answer by word-matching, without the need of multi-hop reasoning, as observed by Jiang and Bansal (2019). Effect of reasoning types. When we train the model with only the Text-to-Table questions (A5), the model achieves 47.9 F1 for In-Table questions and nearly zero performance for In-Passage questions. However, training with only the Table-to-Text questions (A4) also benefits the In-Table questions (15.2 F1). We believe the reason is that the information in the text can also answer some In-Table questions. Using both reasoning types (A6), the model improves on average by 8.6 F1 compared with the models using a single reasoning type (A4, A5). This shows that it is beneficial to train the multi-hop QA model with diverse reasoning chains. Effect of question filtration. Question filtration also helps to train a better QA model, leading to a +2.3 F1 for HybridQA and +1.1 F1 for HotpotQA. We find that the GPT-2 based model can filter out most ungrammatical questions but would keep valid yet unnatural questions such as Where was the event that is held in 2016 held?. We then explore MQA-QG's effectiveness in the few-shot learning setting where only a few human-labeled ( q, a ) pairs are available. We first train the unsupervised QA model based on the training data generated by our best model. Then we finetune the model with limited human-labeled data. The blue line in Figure 5(a) and Figure 5(b) shows the F1 scores with different numbers of labeled training data for HybridQA and HotpotQA, respectively. We compare this with training the QA model directly on the human-labeled data without unsupervised QA pretraining (grey lines in Figure 5). With progressively larger training dataset sizes, our model performs consistently better than the Figure 6: Question type distribution for our generated dataset and the human-labeled dataset for HybridQA. model without unsupervised pretraining for both two datasets. The performance improvement is especially prominent in very data-poor regimes; for example, our approach achieves 69.3 F1 with only 100 labeled examples in HotpotQA, compared with 21.4 F1 without unsupervised pretraining (47.9 ab-solute gain). The results show pretraining QA with MQA-QG greatly reduce the demand for human-annotated data. It can be used to provide a warm start for online learning QA system in which training data are quite limited for a new domain. Although the generated questions are used to optimize for downstream QA performance, it is still instructive to examine the output QA pairs to better understand our system's advantages and limitations. In Figure 6, we plot the question type distribution for both the human-labeled dataset and the generated data for HybridQA. We find that the two datasets have a similar question type distribution, where What questions constitute the major type. However, our model generates more When and Where questions but fewer Which questions. This is because the two reasoning graphs we apply for HybridQA are bridge-type questions while Which questions mostly compare. Table 6 shows representative examples generated by our model. Most questions are fluent and exhibit encouraging language variety, such as Examples 2, 3, 5. Our model also shows almost no sign of semantic drift, meaning most of the questions are valid despite sometimes being unnatural. The two major deficiencies are inaccurate references (in red) and redundancies (in blue), shown in Examples 1, 4, 6. This can be addressed by incorporating minimal supervision to guide the fusion process; i.e. , more flexible paraphrasing in fusion. As discussed in Section 3.3, to generated more natural-looking questions, we attempted to train a BART-based question paraphrasing model to paraphrase each generated question. We finetune the pretrained BART model on the Quora Question Paraphrasing dataset 1 , which contains over 100,000 question pairs with equivalent semantic meaning. The evaluation results are shown in Table 7. Surprisingly, we observe a performance drop for both 1 https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs the HybridQA and the HotpotQA dataset, with a 4 . 3 and 4 . 8 decrease in F1, respectively. We observe that paraphrasing indeed produces more fluent questions by rewriting the redundancy parts of the original questions into more concise expression. However, paraphrasing introduces the se-mantic drift problem, i.e. , the paraphrased question changes the semantic meaning of the original question. We believe this severally hurts the QA performance because it produces noisy samples with inconsistent question and answer. Therefore, we argue that in unsupervised multi-hop QA, semantic faithfulness is more important than fluency for the generated questions. This explains why we design hand-crafted reasoning graphs to ensure the semantic faithfulness. However, how to generate fluent human-like questions while keeping semantic faithfulness is an important future direction. In this work, we study unsupervised multi-hop QA and propose a novel framework MQA-QG to generate multi-hop questions via composing reasoning graphs built upon basic operators. The experiments show that our model can generate human-like questions that help to train a well-performing multi-hop QA model in both the unsupervised and the few-shot learning setting. Further work is required to include more flexible paraphrasing at the fusion stage. We can also design more reasoning graphs and operators to generate more complex questions and support more input modalities. This research is supported by the National Research Foundation, Singapore under its International Research Centres in Singapore Funding Initiative. UCSB authors are not supported by any of the above projects.
abstain objective objective abstain result result other abstain abstain abstain abstain abstain abstain objective method other objective abstain method abstain method abstain abstain method abstain method abstain abstain abstain result result abstain abstain objective objective result other other other other other other other other objective other other other other other other other other other other objective abstain abstain method method method abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain result abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain result abstain result abstain abstain abstain abstain abstain other abstain abstain result abstain abstain abstain abstain abstain objective result abstain method other other