Models pre-trained with a language modeling objective possess ample world knowledge and language skills, but are known to struggle in tasks that require reasoning. In this work, we propose to leverage semi-structured tables, and automatically generate at scale question-paragraph pairs, where answering the question requires reasoning over multiple facts in the paragraph. We add a pre-training step over this synthetic data, which includes examples that require 16 different reasoning skills such as number comparison, conjunction, and fact composition. To improve data efficiency, we sample examples from reasoning skills where the model currently errs. We evaluate our approach on three reasoning-focused reading comprehension datasets, and show that our model, PReasM, substantially outperforms T5, a popular pre-trained encoder-decoder model. Moreover, sampling examples based on model errors leads to faster training and higher performance. Large pre-trained language models (LMs) (Devlin et al., 2019; Liu et al., 2019; Brown et al., 2020) have become the backbone of natural language processing in recent years. However, recent work has shown that they struggle in performing symbolic reasoning operations, such as composition or conjunction of facts (Talmor et al., 2019, 2020), numerical operations (Wallace et al., 2019; Hidey et al., 2020), and quantification (Warstadt et al., 2019), without substantial amounts of additional data. Past work on improving reasoning in pre-trained models has taken two flavors: (a) adding specialized components for specific skills, like numerical and temporal reasoning (Ran et al., 2019; Gupta et al., 2020a; Khot et al., 2021; Chen et al., 2020a), or (b) generating synthetic examples at scale, for example, by using grammars or templates (Rozen  Work done while working at the Allen Institute for Artificial Intelligence. et al., 2019; Zhao et al., 2019; Andreas, 2020; Asai and Hajishirzi, 2020; Campagna et al., 2020), and question generation models (Alberti et al., 2019; Puri et al., 2020; Bartolo et al., 2021). In this work, we take the latter approach and argue that semi-structured tables are a valuable resource for automatic generation of training data that can endow LMs with reasoning skills. Tables can be crawled from the web at scale, and cover a wide range of domains and topics. Moreover, their structured nature makes them amenable to automatic processes of data generation. Specifically, given a table, we use templates to generate reading comprehension (RC) examples, that is, question-context-answer triplets, where answering the question requires diverse types of reasoning over facts mentioned in the context. Fig. 1 shows an example table, and three generated question-context-answer examples, which require fact composition, number comparison, and computing a date difference. Unlike prior work where semi-structured data was used for reasoning over tables or knowledge-bases (Eisenschlos et al., 2020; Yin et al., 2020; Herzig et al., 2020; Yu et al., 2021), here we harness tables to allow LMs to reason over text directly. Fig. 2 provides an overview of our approach. We generate data by crawling tables from Wikipedia, and applying 16 different example generators (EGs) on each table. Each EG corresponds to a particular reasoning skill (composition, numerical comparison, see Table 1 for full list), and comprises a small set of question templates. Variables in the templates are filled with content from the table, and the structure of the table allows to compute the answer automatically. The context is a list of facts generated from the table that contain facts required for answering the question as well as distractor facts. We add a pre-training step over this generated data, where we perform multi-task training over the 16 task corresponding to the EGs. Since each EG can generate vast numbers of examples, it is important to focus training on reasoning skills that the model lacks. Thus, we use error-driven sampling (Gottumukkala et al., 2020) to construct training batches, where most examples are sampled from EGs that the model currently struggles with. We fine tune our P re-traind for Reas oning M odel, PReasM, on three RC datasets that require reasoning: DROP (Dua et al., 2019), IIRC (Fergu-son et al., 2020), and MMQA (Talmor et al., 2021). PReasM outperforms the original pre-trained T5 (Raffel et al., 2020) model by significant margins: 7 . 6 , 4 . 1 , and 1 . 2 F 1 points, respectively. Our results set a new state-of-the-art on MMQA and are the best results on IIRC for models where the retriever and reader are trained separately. Our analysis shows that PReasM leads to improvements of up to 40 F 1 points on specific question types, such as computing the difference between two dates, without causing a drop in other question types. In conclusion, our results suggest that tables are a viable and untapped source of information for automatically generating large amounts of data that can be used to endow LMs with skills that are not captured using current pre-training approaches. Our code, data, and models are publicly available and can be downloaded from https://github. com/oriyor/turning_tables . Our goal is to train a RC model that given a question q and textual context c returns an answer a , given a training set D = { ( q i , c i , a i ) } Ni =1 . We focus on questions that require reasoning over the context, e.g., composing two facts. To endow LMs with reasoning skills, we want to generate a large synthetic training set D syn = { ( q j , c j , a j ) } Mj =1 ( M  N ) from semi-structured tables, before fine-tuning on a target dataset. We use tables from English Wikipedia 1 to generate D syn . English Wikipedia includes millions of tables with high lexical and domain diversity (Fe-tahu et al., 2019; Chen et al., 2020b; Gupta et al., 2020b; Talmor et al., 2021; Nan et al., 2021; Neer-aja et al., 2021a). We extract from Wikipedia all tables T that have at least two columns and 10-25 rows, resulting in more than 700K tables. Then, we annotate all table columns with their semantic type ( STRING , NUMBER , or DATE ), which allows us to generate questions that involve manipulating numbers and dates. Details on the process of column annotation are in A.1. The core of the generation process are the example generators (EGs), each corresponding to a reasoning skill (Table 1). Each example generator g  G is a function that takes a table t  T and randomly samples at most ten ( q, c, a ) triplets from the set of all possible triplets, where (i) q is a question is pseudo-language, (ii) c is the context, i.e., a list of facts extracted from t that includes the gold facts necessary for answering q and distractor facts , all phrased in pseudo-language, and (iii) a is the answer. Overall, the synthetic training set is D syn = (cid:83) t T (cid:83) g G g ( t ) . EGs generate examples in the following way. Each EG is associated with one or more question templates, which differ in their surface phrasing. 2 Templates contain typed variables that are instantiated with content from the table (see all variables in Table 1). Column and value variables are indexed to specify that the variable val:i must be instantiated by a value from the column col:i . Instantiating all variables results in the question 2 We also experimented with using just one question template per EG and observed very similar downstream results. q and the template allows us to programmatically compute the answer a . E.g., in the question from Fig. 1: In League Cup of 199091 Chelsea F.C. season, Which Round had a higher Attendance: QF or QFR? the answer a can be found by finding the rows with the values QF and QFR in the column Round , and returning the value that has a higher number in the column Attendance . The context c is generated from the content necessary for answering the question, which can be identified using the instantiated question template. Facts generally have the form The col:1 when the col:2 was val:2 was val:1 . E.g., to answer the question above, we generate the gold facts The Attendance when the Round was QF was 34,178 , and The Attendance when the Round was QFR was 33,861 . We also generate distractors by generating facts from rows or columns that are not relevant for the question, e.g., The Attendance when the Round was R4 was 9,789 . Data generation yields 4.8M questions from over 176K tables and 130K pages. Table 2 contains examples for generated ( q, c, a ) triplets, including the full context c . Table 3 shows the number of generated examples for each EG. The number of distinct words is large (850K), illustrating the wide coverage and high lexical diversity of our approach. Moreover, generated examples have diverse answer types, which include text spans ( 43 . 2% ), yes/no questions ( 31 . 6% ), numeric ( 15 . 8% ), and date answers ( 9 . 4% ). In addition, our questions cover a wide range of domains including popular culture, politics and science. Tables cover more than 2,500 different Wikipedia categories, with 150 categories covering 80% of the data. Fig. 3 presents the most common categories of the Wikipedia pages from which we scraped our tables. Since our EGs generate large quantities of examples, one can think of each EG as providing an infinite stream of examples. In this setup, a natural question is how to construct training batches such that the model learns the required skills as quickly Figure 3: The most frequent categories of our Wikipedia pages and their frequency. as possible. After briefly describing our model, we will detail our training framework, where we sample examples from EGs in an error-driven manner. Model We use a standard encoder-decoder architecture (Raffel et al., 2020; Lewis et al., 2020). Given a training example ( q, c, a ) , the model takes as input the sequence of tokens  q [SEP] c ', and the task is to autoregressively decode the answer a token-by-token. We train to maximize the maximum likelihood objective log P ( a | q, c ) . Given a pre-trained LM, we add another pretraining step, where we multi-task over a set of tasks S , each task corresponding to examples generated from one EG. Similar to past work (Yogatama et al., 2019; Geva et al., 2020), to avoid catas-trophic forgetting (Kirkpatrick et al., 2016) of the 6019 language skills, we sample batches from the original pre-training task with probability  = 0 . 5 . Past work (Gottumukkala et al., 2020) has shown that heterogeneous batching , i.e., having examples from all tasks in each batch, leads to better performance compared to having entire batches from a single task. We follow this practice, and in every batch sample examples from every task according to a probability distribution P tasks  R |S| . The main question is how to determine the distribution P tasks , which we turn to next. Uniform sampling Past work (Khashabi et al., 2020; Raffel et al., 2020; Wang et al., 2020) used uniform sampling, where the probability to sample from a task s is P tasks ( s ) = 1 |S| , as a-priori all tasks are equally important. Some approaches also sample examples in proportion to the size of the training set (Raffel et al., 2020; Wang et al., 2020). This is not applicable in our case, where we assume an infinite stream of examples for every task, and make no assumptions on the distribution over reasoning skills in the downstream test set. Error sampling Recent work (Sharma et al., 2018; Gottumukkala et al., 2020) proposed to construct P tasks based on model errors, where one over-samples tasks with higher errors. More formally, let Ceil ( s ) be an estimate of the maximum accuracy achievable on a task s , and Acc ( s ) be the current model accuracy for task s on an held-out set. We define ( s ) = Ceil ( s )  Acc ( s ) and P tasks ( s ) = ( s ) (cid:80) s S ( s  ) . The distribution P tasks of a task is updated every time we evaluate the current model on the held-out data. In our setup, since the data is synthetic and abundant, we assume that the ceiling accuracy for all tasks is 1 . 0 , and hence: ( s ) = 1 . 0  Acc ( s ) . Similar to Gottumukkala et al. (2020), we use accuracy over a held-out set rather than the training loss, as this corresponds directly to our target metric. Momentum sampling A potential issue with error sampling, is that if the error rate on a task is high, the model will spend most of its time on that task at the expense of other tasks, which may lead to low data efficiency. To remedy this, we introduce momentum sampling , a sampling strategy that Algorithm 1 Momentum Sampling( w , t, , k ) Input: windows size w , training time t , minimum share of examples per task  , smoothing factor k . samples from a task in proportion to its rate of improvement , putting most probability mass on skills that are improving rapidly. Alg. 1 provides the details of momentum sampling. Let t denote the index of a checkpoint evaluated on the held-out set, let w be a window size, and Acc s ( i ) be the held-out accuracy of checkpoint i on task s . We estimate model accuracy on a task s at the beginning and end of the window, and sample examples in proportion to the difference 3 in accuracy during that window. To smooth out accuracy fluctuations in adjacent checkpoints, we estimate accuracy as an average of k model checkpoints. During the first w checkpoint evaluations, we simply use uniform sampling. Momentum sampling has several theoretical benefits over error sampling. First, it does not assume anything on the ceiling accuracy of a task. Second, when all tasks converge to their ceiling accuracy, momentum sampling converges to uniform sampling, unlike error sampling, which will over-sample from tasks for which Ceil ( s ) is low. This is useful in cases where the variance of Ceil ( s ) is high across tasks. On the other hand, momentum sampling requires a warm-up of w steps, and might under-sample from tasks that train slowly. In A.2., we describe two synthetic experiments where momentum sampling clearly outperforms error sampling. However, we do not observe an advantage for momentum sampling in our experiments in 5, and leave further investigation of momentum sampling to future work. Baselines Our main baseline is T5 (Raffel et al., 2020), a popular pre-trained encoder-decoder model, which we fine-tune on the downstream 3 We use the difference in performance and not the gain to account for cases of sudden drops in performance. datasets. We experiment with Base and Large size models. For each dataset, we compare to the relevant state-of-the-art model. Our pre-trained for reasoning model, PReasM, is a T5 model with another pre-training step on D syn . We experiment with uniform sampling ( PReasM-Uni ), error sampling ( PReasM-Err ), and momentum sampling ( PReasM-Moment ) strategies. DROP (Dua et al., 2019) is a RC dataset with questions that require numeric reasoning. As an additional baseline, we also compare to GenBERT (Geva et al., 2020), which similar to our approach injects numerical skills by automatically generating synthetic data from a grammar. IIRC (Ferguson et al., 2020) is a QA dataset, where annotators were given a single Wikipedia paragraph, and were asked to author questions that depend on that paragraph, but also on other paragraphs linked from the input paragraph. This resulted in questions that require discrete temporal ( 28% ) or numeric ( 11% ) reasoning. In addition, 30% of the questions are unanswerable. We experiment with IIRC in two settings: (a) Oracle , where the model is given the gold context, reducing the problem to RC, where we can apply our models. (b) Retrieval , where we use the im-proved pipelineintroduced by Ni et al. (2021) to retrieve the context, and replace the NumNet+ (Base) reader (Ran et al., 2019) used by the authors (which has specialized architecture) with T5/PReasM. MMQA (Talmor et al., 2021) is a QA dataset, where the input is a question and a context that consists of a table, multiple text paragraphs, and multiple images, and the model must reason over a subset of the input modalities to answer the question. 4 We chose to use MMQA as it has many questions that involve a conjunction of facts, an operation that is largely missing from other datasets. Moreover, a large fractions of the questions can be answered by reasoning over the text and table only. Since T5/PReasM cannot handle images or very long contexts, we construct a pipeline that automatically directs some MMQA questions to T5/PReasM, and uses the original Implicit-Decomp baseline from Talmor et al. (2021) elsewhere. Our pipeline includes two classifiers, the first determines whether a question requires reasoning over 4 We removed tables that appear in the MMQA development and test sets from D syn . an image, and the second classifies whether a text paragraph is necessary to answer a question. Again, we experiment with an oracle and retrieval setting, such that in the oracle setting our model is presented with the gold paragraphs. We provide the full description of this pipeline in A.4. Evaluation metrics For all datasets, we use the official scripts for computing F 1 and EM, which compare the gold and predicted list of answers. Table 4 presents the results of our large models over all datasets, also in comparison to current state-of-the-art. We observe that PReasM substantially improves performance compared to T5 in all conditions, improving on the test set by 7 . 6 , 7 . 9 , 4 . 1 , and 1 . 2 F 1 points on DROP, IIRCoracle, IIRC, and MMQA respectively. 5 We obtain new state-of-the-art results on MMQA and IIRCoracle. On IIRC, we improve performance when using the same retriever (Pipeline) and replacing the Num-Net+ reader with PReasM. 6 On DROP, specialized architectures for handling numbers still substantially outperform both T5 and PReasM. Table 5 shows the effect of different sampling strategies. Error sampling and momentum sampling generally outperform uniform sampling, but there is no clear advantage to momentum sampling compared to error sampling. We further analyze the effect of sampling methods in 5.2. We now look at performance on different answer types across datasets, where PReasM leads to dramatic improvements on some types, while maintaining similar performance on other types. DROP Table 6 breaks down performance based on answer types: PReasM outperforms T5 across 5 To verify that the gains of PReasM over T5 are not due to knowledge memorized from D syn , we trained T5 and PReasM to generate the answer given the question only (without con-text). We found that the performance of T5 and PReasM is nearly identical in this setup. 6 We report the numbers from Ni et al. (2021) ( 45 . 8 / 44 . 3 F 1 on the development/test sets). To fairly compare with the NumNet+ reader, we got the retrieved paragraphs for the Pipeline model through personal communication. However, results on these paragraphs was lower than reported in the paper: 45 . 5 / 42 . 8 F 1 . The reported results of our models are with this slightly worse retriever, but still outperform the performance of NumNet+ (Pipeline) from the original paper. the board for all model sizes and answer types. PReasM-Base outperforms GenBERT on 3 of 4 answer types. The high performance of GenBERT on Number questions can be explained by: (a) GenBERT uses digit tokenization which improves arithmetic reasoning (Thawani et al., 2021), and (b) training on multiple numerical reasoning templates. IIRC Table 7 breaks down performance based on answer types. PReasM outperforms T5 in the oracle setup by roughly 8 points for both Base and Large models, and by 2 . 6 4 points in the retrieval setup. Improvements are mostly due to cases when the answer is a numerical Value , where PReasM outperforms T5 by 39 . 1 and 40 . 3 F 1 points in Base and Large models (oracle setup). Comparing PReasM-Base to NumNet+, PReasM outperforms NumNet+ on None , Span and Binary questions, but lags behind on Value questions, where NumNet+ uses specialized architecture. Overall, PReasM-Large improves state-of-the-art in the oracle setup by 4 . 7 F 1 points. In the retrieval setting, PReasM outperforms NumNet+ (Pipeline) by 4 . 2 and 0 . 8 F 1 points on the development and test sets, respectively (see Table 4). MMQA Table 8 breaks down performance based on reasoning skills (annotated per example in MMQA). PReasM outperforms T5 in both the oracle and retrieval setting, and for both model sizes. The main cause for improvement are comparison questions, where PReasM outperforms T5 by 19 and 11 . 7 F 1 points on Base and Large models. PReasM outperforms T5 on conjunction questions in Base models, and yes/no questions in all settings. Interestingly, T5 is equipped with decent composition skills, without any specialized pre-training. Compared to Implicit-Decomp , although Implicit-Decomp outperforms our models on questions that require hopping between two table columns and aggregations, PReasM outperforms Implicit-Decomp in all other cases. When considering only questions that require reasoning over text and tables, PReasM-Large improves F 1 by 16 . 1 points, from 62 . 3 to 78 . 4 . Fig. 4 shows statistics on the performance of PReasM on different tasks in D syn during training. The average accuracy across all tasks at the end of training is high  almost 98.0 F 1 . PReasM reaches high performance on all tasks, where the lowest-performing tasks are arithmetic addition' ( 91 . 1 ) and date difference' ( 94 . 7 ). On those tasks, the advantage of error-driven sampling is evident, and it outperforms uniform sampling by as much as 4 points. Zooming-in on the learning curve, momentum 6022 Model Oracle ColumnHop Text Composition Comparison Conjunction Yes/No Aggregate Total T5-Base  81.7 75.2 67.0 61.8 74.1 76.9 27.3 71.9 PReasM-Base  80.8 75.7 66.3 80.8 80.8 83.1 36.4 74.3 T5-Large  82.6 79.8 71.8 69.3 83.0 83.1 27.3 76.8 PReasM-Large  84.0 79.7 71.9 81.0 82.3 93.8 36.4 78.4 T5-Base  85.2 82.1 74.6 63.3 77.4 80.0 27.3 77.9 PReasM-Base  86.9 80.0 75.4 84.1 82.6 89.2 36.4 79.9 T5-Large  88.2 85.9 79.4 74.1 83.2 83.1 36.4 82.7 PReasM-Large  87.8 85.6 79.8 83.6 82.3 90.8 45.5 83.8 Implicit-Decomp  96.6 57.1 53.2 78.4 68.1 76.9 59.1 62.3 Table 8: Development F 1 on MMQA with reasoning type breakdown on the development set. and error sampling learn reasoning skills a lot faster than uniform sampling. Looking at the entropy of P tasks sheds light on the difference between error sampling and momentum sampling. Error sampling puts most probability mass on the lowest-performing task (arithmetic addition), and thus its entropy over tasks is roughly constant from a certain point. Conversely, momentum sampling puts a lot of probability mass on tasks that are improving quickly at the beginning, but as improvements plateau, it converges towards uniform sampling. Fig. 5 and Table 11 (in the Appendix) show the results for T5 and PReasM on D syn . The results for T5 were obtained by training in a few-shot manner on 32 examples for 200 steps, as suggested in Ram et al. (2021). T5-Large outperforms T5-Base on most tasks, suggesting that larger models are able to learn reasoning skills faster. On tasks such as date difference and arithmetic addition, the results for T5-Large are low, at around 10 F 1 . Our PReasM models significantly outperform T5 on all tasks. Reasoning skills in DROP To check which reasoning skills PReasM has, we use a proposed split of a subset of DROP to reasoning skills (Gupta et al., 2020a). Table 9 presents the F 1 for our best PReasM and T5 models, as well as the F 1 from Question Type NMN T5-PReasMT5-PReasMBase Base Large Large Date-Compare 82.6 86.4 87.5 87.6 89.9 Date-Difference 75.4 19.6 78.9 45.4 80.4 Number-Compare 92.7 91.3 95.2 97.3 98.5 Extract-Number 86.1 91.8 94.9 92.1 95.1 Count 55.7 80.1 86.7 86.7 89.2 Extract-Argument 69.7 87.6 86.2 90.5 92.1 Table 9: F 1 on a previously-proposed split of a subset of the development set of DROP to reasoning skills. the neural module network (NMN) used in Gupta et al. (2020a). NMN was trained only on a subset of the original DROP dataset. When comparing to T5, PReasM dramatically improves performance on Date-Difference, and also leads to sizable gains in Number-Compare, Extract-Number and Count. Accuracy vs. training cost trade-off We evaluate PReasM-Base models on DROP and IIRCoracle as we vary the number of pre-training steps on D syn (Fig. 6). Most of the improvement happens in the first 100K steps, and error-driven sampling outperforms uniform sampling throughout training. Error sampling outperforms momentum sampling in the latter part of training. A possible reason is that the reasoning skills in the downstream tasks are correlated with the harder tasks during pre-training (arithmetic addition and date difference). This provides an advantage for error 6023 Figure 5: F 1 for each task in D syn , for T5 and PReasM on the held-out evaluation set. sampling, since it will focus on these tasks even if the improvement during pre-training is small. Template-based data generation has been previously used for data augmentation, for example to inject numerical skills (Geva et al., 2020), and to improve consistency (Asai and Hajishirzi, 2020), and zero-shot accuracy (Zhao et al., 2019). In addition, templates were used for dataset construction (Talmor and Berant, 2018; Clark et al., 2020; Thorne et al., 2021), and to analyse model generalization (Rozen et al., 2019). In this work, we automatically generate examples by instantiating templates using structured data. Since our method relies solely on tables as input, it is highly scalable, has rich lexical diversity, and can be easily extended to new skills and domains. Recently, Thorne et al. (2021) introduced the WIKINLDB dataset, which includes queries that require reasoning over a set of textual facts. Queries are instantiated with values from a knowledge graph (KG), and facts are generated by a LM. Unlike this work, WIKINLDB is focused on evaluating reasoning skills. We, on the other hand, show that generated examples can be used to endow a pre-trained LM with new reasoning skills. Moreover, tables are much easier to collect at scale compared to KGs, which tend to have limited coverage. Data augmentation techniques have been extensively explored in RC, QA, and dialogue (Feng et al., 2021; Talmor and Berant, 2019; Khashabi et al., 2020; Alberti et al., 2019; Puri et al., 2020; Bartolo et al., 2021). Here, we focus on tables as a valuable source for data generation. Pre-training over tables has focused in the past on reasoning over tables and knowledge-bases (Eisen-schlos et al., 2020; Yin et al., 2020; Herzig et al., 2020; Mller et al., 2021; Yu et al., 2021; Neer-aja et al., 2021b). Here, we use pre-training over tables to improve reasoning over text . We leave evaluation on tasks beyond RC to future work. Error-driven sampling has been considered in the past in the context of active learning (Sharma et al., 2018), reinforcement learning (Graves et al., 2017; Glover and Hokamp, 2019; Xu et al., 2019), transfer learning (Zhang et al., 2020; Pilault et al., 2021), and distributionally robust optimization (Oren et al., 2019; Sagawa et al., 2020), where the goal is to perform well over a family of distributions. Similar to Gottumukkala et al. (2020), we compute heterogeneous batches based on error rates, and show that this improves efficiency and performance. We propose semi-structured tables as a valuable resource for generating examples that can endow pre-trained language models with reasoning skills. We generate 5M examples that correspond to 16 reasoning skills from Wikipedia tables and add a pretraining step over this data. To improve data efficiency we use error-driven sampling, which focuses training on reasoning skills that the model currently lacks. We evaluate our model, PReasM, on three reasoning-focused RC datasets and show that it leads to substantial improvements in all cases. We thank Elad Segal, Uri Shaham, Tomer Wolfson, and Ankit Gupta for their useful comments and James Ferguson, Ansong Ni and Matt Gardner for their help with the IIRC dataset. This research was partially supported by The Yandex Initiative for Machine Learning, and the European Research Council (ERC) under the European Union Horizons 2020 research and innovation programme (grant ERC DELPHI 802800).
abstain objective method result result abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain method method method abstain abstain abstain method abstain method abstain abstain abstain abstain abstain objective result result other abstain abstain method method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method method method abstain method method abstain abstain method method abstain abstain method abstain abstain abstain method abstain abstain method method method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method method method method abstain method abstain abstain abstain method method method abstain method abstain method abstain abstain method method method result result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain result abstain other abstain abstain abstain method abstain abstain abstain abstain abstain other other method objective other other method objective other other method other abstain abstain other abstain objective method result result other other