We introduce S2ORC, 1 a large corpus of 81.1M English-language academic papers spanning many academic disciplines. The corpus consists of rich metadata, paper abstracts, resolved bibliographic references, as well as structured full text for 8.1M open access papers. Full text is annotated with automatically-detected inline mentions of citations, figures, and tables, each linked to their corresponding paper objects. In S2ORC, we aggregate papers from hundreds of academic publishers and digital archives into a unified source, and create the largest publicly-available collection of machine-readable academic text to date. We hope this resource will facilitate research and development of tools and tasks for text mining over academic text. Academic papers are an increasingly important textual domain for natural language processing (NLP) research. Aside from capturing valuable knowledge from humankind's collective research efforts, academic papers exhibit many interesting characteristics  thousands of words organized into sections, objects such as tables, figures and equations, frequent inline references to these objects, footnotes, other papers, and more. Different types of resources have been used to support research over academic papers. Citation graphs like AMiner's Open Academic Graph (Tang et al., 2008), the Microsoft Academic Graph (MAG) (Shen et al., 2018), and the Semantic Scholar literature graph (Ammar et al., 2018), have had widespread application in bibliomet-rics, science-of-science, information retrieval, and network analysis. Digital archives like arXiv, 2  denotes equal contribution 1 Instructions for access to the data and model are available at https://github.com/allenai/s2orc/ . 2 https://arxiv.org Figure 1: Inline citations and references to figures and tables are annotated in S2ORC's structured full text. PubMed Central, 3 CiteSeerX (Giles et al., 1998), 4 and the ACL Anthology (Bird et al., 2008), 5 are popular resources for deriving large text corpora for summarization and language modeling or, with further annotation, development of datasets for tasks like entity extraction, text classification, parsing, and discourse analysis. We focus on bibliometrically-enhanced derivations of these corpora, such as the ACL Anthology Network (AAN) (Radev et al., 2009) 6 derived from the ACL Anthology, RefSeer (Huang et al., 2015) derived from CiteSeerX, and Saier and Farber (2019) derived from arXiv, which combine useful aspects of citation graphs and raw text corpora. These resources provide citation mentions linked to paper identifiers in their corresponding digital archives, such as the ACL Anthology and CiteSeerX, or to nodes in citation graphs such as MAG, enabling new forms of cross-paper discourse analysis (e.g., studying how or why papers are related). 3 https://www.ncbi.nlm.nih.gov/pmc 4 https://citeseerx.ist.psu.edu 5 https://www.aclweb.org/anthology 6 http://aan.how/ Corpus Papers w/ body text Citation contexts References to tables / figures / equations Linked to graph Academic disciplines S2ORC (PDF-parse) 8.1M full text yes S2ORC (full) multi S2ORC (LATE X-parse) 1.5M full text yes S2ORC (full) physics, math, CS PubMed Central (OA) 2.6M full text yes PubMed bio, med AAN (Radev et al., 2009) 25k full text no ACL Anthology comp ling Saier and Farber (2019)  1.0M snippets no MAG physics, math, CS RefSeer (Huang et al., 2015) 1.0M snippets no CiteSeerX multi Table 1: A comparison of S2ORC with other publicly-available academic text corpora. Yet, existing corpora are not without their limitations. Some cover a small number of papers (e.g. AAN), are domain-specific (e.g. AAN, PubMed Central, Saier and Farber (2019)), or may not provide usable full text (e.g. Saier and Farber (2019) and RefSeer). To address these issues, we introduce S2ORC, 7 the Semantic Scholar 8 Open Research Corpus, a large publicly-available collection of 81.1M academic papers covering dozens of academic disciplines. Each paper is associated with metadata and abstracts aggregated from hundreds of trusted sources such as academic publishers and literature archives like PubMed and arXiv. Notably, we release structured, machine-readable full text extracted from PDFs for 8.1M papers which we've identified as having open access status. S2ORC full text preserves meaningful structure, e.g., paragraph breaks, section headers, inline citation mentions, references to tables and figures, and resolved citation links to other papers. Additionally, we provide 1.5M full text LATEX parses from which we have extracted, in addition to citations and references, the source text of tables and mathematical formulas. As shown in Table 1, S2ORC provides substantially more structured full text papers and covers a more diverse set of academic disciplines than other resources. pronounced stork 8 The papers included in S2ORC are a curated subset of the papers in the Semantic Scholar literature graph (Ammar et al., 2018) that focuses only on English-language papers with abstracts or full text available. See 2.5 for details on filtering through Semantic Scholar papers. In this paper, we describe the construction of S2ORC (2). We provide summary statistics of the corpus (3) and evaluate the data quality (4). We then evaluate a BERT model pretrained on S2ORC (5), and discuss potential applications to a variety of NLP and analysis tasks over academic text (6). Finally, we compare S2ORC with other publicly-available academic text corpora (7). S2ORC is constructed using data from the Semantic Scholar literature corpus (Ammar et al., 2018). Papers in Semantic Scholar are derived from numerous sources: obtained directly from publishers, from resources such as MAG, from various archives such as arXiv or PubMed, or crawled from the open Internet. Semantic Scholar clusters these papers based on title similarity and DOI overlap, resulting in an initial set of approximately 200M paper clusters. To construct S2ORC, we must overcome challenges in (i) paper metadata aggregation, (ii) identifying open access publications, and (iii) clustering papers, in addition to identifying, extracting, and cleaning the full text and bibliometric annotations associated with each paper. The pipeline for creating S2ORC is: 1) Process PDFs and LATEX sources to derive metadata, clean full text, inline citations and references, and bibliography entries, 2) Select the best metadata and full text parses for each paper cluster, 3) Filter paper clusters with insufficient metadata or content, and 4) Resolve bibliography links between paper clusters in the corpus. Details for these steps are provided below. See Appendix A for definitions of terminology. The output of this pipeline is visualized in Figure 1. 2.1 Processing PDFs We process PDFs from the Semantic Scholar corpus using SCIENCEPARSE v3.0.0 9 and GROBID v0.5.5 10 (Lopez, 2009). Our processing pipeline is described below. Selecting PDFs We remove PDFs which are less likely to be academic papers. SCIENCEPARSE and GROBID are not optimized for processing non-paper academic documents such as dissertations, reports, slides, etc., and this filtering step is necessary to increase output data quality. See Appendix B for filter details. There are around 31.3M PDFs associated with approximately 200M initial paper clusters, and 30.5M PDFs are selected for processing based on these filtering criteria. Extracting structured data from PDFs We use SCIENCEPARSE to extract title and authors from each PDF. 11 We then use GROBID to process each PDF. From the XML output of GROBID , we extract (i) metadata such as title, authors, and abstract, (ii) paragraphs from the body text organized under section headings, (iii) figure and table captions, (iv) equations, table content, headers, and footers, which we remove from the body text, (v) inline citations in the abstract and body text, (vi) parsed bibliography entries with title, authors, year, and venue identified, and (vi) links between inline citation mentions and their corresponding bibliography entries. Postprocessing GROBID output We postprocess GROBID output using regular expressions to classify the parenthetical citation style of a paper as BRACKET (e.g. [2]), NAME-YEAR (e.g. ABC, 2019), or OTHER (superscripts and other mixed styles). We focus on addressing two types of common errors in GROBID 's inline citation extractions: (i) false positives resulting from superscripts or equation references being recognized as 9 https://github.com/allenai/science-parse 10 https://github.com/kermitt2/grobid 11 Our evaluations suggest SCIENCEPARSE outperforms GROBID for title and author extraction. inline citations in papers with BRACKET -style citations, and (ii) false negatives resulting from an inability to expand bracket citation ranges (e.g. [3]-[5] should be expanded to [3], [4], [5] before linking). False positives are detected using regular expressions and removed from GROBID output. Bracket citation ranges are manually expanded and linked to their corresponding bibliography entries. The resulting parses are expressed in JSON format. 12 2.2 Processing LATEX source LATEX document source is available for a majority of arXiv submissions, and where available, are used to construct a full text parse. We retrieve body text, section headers, figure/table captions, table representations, equations, and inline citations and references directly from LATEX source. Inspired by Saier and Farber (2019), we first convert LATEX source into XML documents and then extract structured information from the XML. Due to direct access to source, the accuracy of citation span, reference, caption, section header, and equation detection is near-perfect. We process 1.5M papers from LATEX source derived from arXiv, all of which are included as part of S2ORC. Surprisingly, due to the diversity of ways in which authors define metadata in LATEX, the quality of metadata extracted from LATEX documents is worse than those extracted from PDF. Therefore, we do not use LATE X-derived metadata for paper clustering or metadata selection. Canonical values for title, authors and other metadata fields are selected from among the papers in a cluster. First, if a cluster contains multiple PDFs, we select one to be canonical. This can occur, for example, in a cluster containing an arXiv preprint and its eventual camera-ready version. We preferentially select PDFs from open access sources and break ties by prioritizing PDFs for which there exist richer publisher-provided metadata (e.g. abstract, year, venue, DOI). If the selected PDF is associated with publisher-provided metadata, we select those publisher-provided metadata fields to be canonical. In cases where publisher-provided metadata is incomplete, we use majority voting to select 12 The S2ORC data format is described at https:// github.com/allenai/s2orc canonical metadata values. We break ties by minimizing the total number of sources from which we select metadata (e.g., if IEEE provides title, authors and abstract, DBLP provides title and authors, and arXiv provides title and abstract, we prioritize selecting IEEE over the union of DBLP and arXiv). S2ORC metadata fields include title, author, year, venue, journal, abstract, and identifiers (DOI, PubMed, PubMed Central (PMC), arXiv, and ACL Anthology). In cases where the title and authors are not provided by any publishers, we derive the values for these fields from the parsed PDF, prioritizing SCIENCEPARSE over GROBID . We further comment on paper clustering as it pertains to metadata selection in Appendix C. We construct the final corpus by assembling clustered paper metadata with GROBID and LATEX parse objects. We associate the GROBID parse with the S2ORC paper object if a valid GROBID parse is produced from the PDF, and the PDF is open access. Open access status is assigned if a paper is derived from arXiv, ACL Anthology, PubMed Central (OA), and/or associated with an open-access DOI in the Unpaywall database. 13 If the PDF is not open access, we only include the bibliography from the GROBID parse in S2ORC. If arXiv LATEX source is available for the paper cluster, we also associate the LATEX parse with the S2ORC paper object. We further filter paper clusters to remove papers with (i) no title, (ii) no authors, (iii) fewer than 100 characters of abstract and body text, and (iv) where English is not the primary language. The first three filters remove papers that provide little value for bibliometric-based or text-based analyses. The English language filter 14 reduces GROBID parsing errors. All filters are applied in series. Subsequently, 95.5M paper clusters are filtered out based on the aforementioned criteria and removed from the corpus. The distribution of filtered papers is given in Table 2. We note that a large number of paper clusters are filtered out; 80.0M of these filtered clusters have no associated publisher-provided abstract or associated PDF and 13 Unpaywall 2019-04-19 data dump 14 We use the cld2 tool for language detection with a threshold of 0.9 over the English language score. do not provide significant value to our dataset in their current state. Although these papers that lack text may be useful as cite-able nodes in S2ORC, they are generally of lower quality and are filtered out of the corpus to improve corpus quality. Each bibliography entry in both GROBID and LATEX parses are linked to the most similar papers in the corpus. For linking, we score each bibliography entry and paper cluster pair using a similarity score computed between their titles. Each title is first normalized (i.e. white spaces stripped, lower-cased, special characters removed) and represented by its character 3-grams. The similarity score S title is computed as the harmonic mean between a Jaccard index and a containment metric: S title = 2  J  C J + C (1) where the Jaccard index J and containment metric C are computed from the n -grams of the two titles N 1 and N 2 as: J = | N 1  N 2 | | N 1  N 2 | C = | N 1  N 2 | min ( | N 1 | , | N 2 | ) For each bibliography entry, the bibliography-paper pair with the highest similarity score above 0.8 is output as the correct link. Otherwise, the bibliography entry remains unlinked. We perform an evaluation of linking performance in 4. The resulting corpus consists of 81.1M papers. Our publisher-provided abstract coverage is 90.4%, or 73.4M papers. Our PDF coverage is 35.6%, or 28.9M papers. These PDFs are processed using the pipeline discussed in 2.1. The Total papers 81.1M Papers w/ PDF 28.9M (35.6%) Papers w/ bibliographies 27.6M (34.1%) Papers w/ GROBID full text 8.1M (10.0%) Papers w/ LaTeX full text 1.5M (1.8%) Papers w/ publisher abstract 73.4M (90.4%) Papers w/ DOIs 52.2M (64.3%) Papers w/ Pubmed IDs 21.5M (26.5%) Papers w/ PMC IDs 4.7M (5.8%) Papers w/ ArXiv IDs 1.7M (2.0%) Papers w/ ACL IDs 42k (0.1%) Table 3: Statistics on paper provenance. vast majority of these PDFs are successfully processed using GROBID , and we extract bibliography entries for 27.6M of the 28.9M PDFs. We identify 8.1M of the 28.9M PDFs as open access (2.4), and we provide full text for all papers in this open access subset. For the 1.5M papers for which LATEX source is available through arXiv, we further obtain and provide LATEX parses (2.2). Using these extracted bibliographies, we resolve a total 380.5M citation links between papers (2.6), 156.5M of which can be tied back to their inline citation mentions in the full text. See Table 3 for more provenance statistics. We provide statistics for the GROBID and LATEX full text parses and bibliography linking in Figure 2: Distribution of papers by Microsoft Academic field of study. Table 4. On average, LATEX parses contain many more paragraphs of body text, because LATEX source files preserve line breaks rather than paragraph breaks. We speculate that differences in bibliography entry and linking counts between the GROBID and LATEX parses are due to a combination of: (i) challenges in LATEX bibliography expansion and parsing, and (ii) differences in bibliography formatting in some math and physics venues (where bibliography entries do not include paper titles, which we depend on for bibliography linking). The distribution of academic disciplines in S2ORC is given in Figure 2 using Microsoft Academic fields of study. Not all papers in S2ORC can be found in Microsoft Academic  those not found are denoted as Unclassified . Approximately 677k papers have more than one primary Microsoft Academic field of study; Figure 2 represents only the top field of study for each paper. To evaluate the quality of our metadata selection, we randomly sample 500 paper clusters, restricting to those with PDFs. Within each sampled cluster, we determine whether the canonical title and authors match the title and authors in the selected canonical PDF. Inline citation detection and bibliography parsing are dependent on GROBID (Lopez, 2009). Ahmad and Afzal (2018) evaluate GROBID for de-Domain Dataset Reference Task SCIBERT S2ORC-S CIBERT BC5CDR Li et al. (2016) NER 90.01 90.41  0.06 JNLPBA Collier and Kim (2004) NER 77.28 77.70  0.25 NCBI-disease Dogan et al. (2014) NER 88.57 88.70  0.52 Biomed EBM-NLP Nye et al. (2018) PICO 72.28 72.35  0.95 GENIA Kim et al. (2003) DEP (LAS) 90.43 90.80  0.19 GENIA Kim et al. (2003) DEP (UAS) 91.99 92.31  0.18 ChemProt Krallinger et al. (2017) REL 83.64 84.59  0.93 SciERC Luan et al. (2018) NER 67.57 68.93  0.19 CS SciERC Luan et al. (2018) REL 79.97 81.77  1.64 ACL-ARC Jurgens et al. (2018) CLS 70.98 68.45  2.47 Biomed & CS SciCite Cohan et al. (2019) CLS 85.49 84.76  0.37 Multi-domain PaperField Beltagy et al. (2019) CLS 65.71 65.99  0.08 Table 5: S2ORC-S CIBERT test results are comparable with reported SCIBERT test results on the set of tasks and datasets from Beltagy et al. (2019), to which we refer the reader for descriptions. tecting inline citations using a corpus of 5k Cite-Seer papers, and found GROBID to have an F1-score of 0.89 on this task. Tkaczyk et al. (2018) report GROBID as the best among 10 out-of-the-box tools for parsing bibliographies, also achieving an F1 of 0.89 in an evaluation corpus of 9.5k papers. We perform an evaluation over 200 randomly sampled papers from S2ORC and found comparable F1-scores for GROBID performance on both tasks. For bibliography linking, we randomly sample S2ORC papers (500 GROBIDPDF parses and 100 LATEX parses) and select one linked bibliography entry from each sampled paper (while avoiding selecting multiple entries linked to the same paper). We determine whether the title and authors in the bibliography entry agree with the title and authors of the linked paper. To demonstrate the suitability of S2ORC for language model pretraining, we train BERT-Base (Devlin et al., 2019) on the parsed full text of S2ORC and show that the resulting model (S2ORC-S CIBERT) performs similarly to SCIBERT (Beltagy et al., 2019) on a diverse suite of scientific NLP tasks and datasets. While SCIBERT is a BERT-Base model also trained on multiple domains of scientific text, key differences in its pretraining corpus and vocabulary and those used for S2ORC-S CIBERT are:  Domain: Beltagy et al. (2019) report a pretraining corpus consisting of 82% biomedical and 18% computer science papers. Our S2ORC pretraining corpus consists of a more balanced distribution of papers across diverse academic disciplines (see Figure 2), such that biomedical (42.7%) and computer science (7.2%) papers only comprise half the corpus. Preprocessing: S2ORC identifies figure captions, table text and captions, headers, footers, and footnotes. We exclude these from the pretraining corpus. We tokenize and sentencize the text using scispaCy (Neumann et al., 2019). We also use heuristic filters to remove ill-formed paragraphs (such as those containing too many symbols). Size: The resulting S2ORC pretraining corpus contains 16.4B tokens, nearly five times larger than the corpus for SCIBERT. Vocab: Following Beltagy et al. (2019), we construct a cased WordPiece (Wu et al., 2016) vocabulary of size 31k using 15% of the S2ORC pretraining corpus. The Jaccard index between the S2ORC-S CIBERT and SCIBERT vocabularies is 0.536. We follow a similar setup to Beltagy et al. (2019) for both pretraining and fine-tuning S2ORC-S CIBERT. Like SCIBERT, S2ORC-S CIBERT is pretrained from scratch using the original BERT code 15 and default BERT-Base configurations on a single TPU v3-8 for one week. Also like SCIBERT, S2ORC-S CIBERT is fine-tuned on all tasks by optimizing a cross entropy loss using Adam (Kingma and Ba, 2014), a linear learning rate decay with 10% warm-up, batch size of 32, and dropout of 0.1. We search over an equal-sized grid of hyperpa-rameters as Beltagy et al. (2019). We fine-tune for 1 to 4 epochs with a maximum learning rate of 1e-5, 2e-5, 3e-5, or 5e-5. For each task, we select the optimal combination of these two hyperparam-eters using the development set and report the corresponding test set results. For details, we refer the reader to SCIBERT code, 16 which we use for all experiments. The results in Table 5 show that S2ORC-S CIBERT outperforms SCIBERT on many tasks despite including a large percentage of data outside of the biomedical and computer science domains. As the pretraining corpus for SCIBERT is not publicly-available, S2ORC can serve as a large pretraining corpus for evaluating and comparing pretraining approaches on academic text. We also release S2ORC-S CIBERT to serve as a baseline for research. S2ORC can be used for many NLP and analysis tasks over academic text. We give a summary of potential applications below. The combination of structured full text annotated with linked inline citations makes S2ORC well-suited for a variety of citation-related text-based tasks. Without any additional supervision, S2ORC can be used directly for both inline (He 15 https://github.com/google-research/ bert 16 https://github.com/allenai/scibert et al., 2010; Duma and Klein, 2014; Jeong et al., 2019) and document-level (Yu et al., 2012; Liu et al., 2015; Bhagavatula et al., 2018) citation recommendation. Among document-level recommenders, S2ORC is well-suited to the setting of Liu et al. (2015), who use inline citation contexts to filter document-level recommendations. include classifying citation intent (Teufel et al., 2006; Jurgens et al., 2018; Cohan et al., 2019), identifying citation sentiment (Athar and Teufel, 2012), identifying meaningful citations (Valen-zuela et al., 2015), extracting key phrases (Caragea et al., 2014), and citation context-based paper summarization (Teufel et al., 2006; Qazvinian and Radev, 2008; Cohan and Goharian, 2015; Mitrovic and Muller, 2015). The models in these papers require labeled citation contexts for training. S2ORC could potentially benefit task performance without additional annotation, for example, by pretraining language models on S2ORC citation contexts before fine-tuning to these tasks. Cohan et al. (2019) find that long citation contexts (beyond sentence boundary) are important for tasks like summarization; the wider citation contexts available in S2ORC could be used to augment existing datasets for document-level tasks. Citation contexts can also be used for the more general tasks of identifying similar papers (Kanakia et al., 2019; Eto, 2019; Haruna et al., 2018; Small, 1973) or bibliometric analysis (Ding et al., 2014; Trujillo and Long, 2018; Asatani et al., 2018). Towards these tasks, the citation contexts in S2ORC can provide insight into how and why papers are cited. We illustrate this by following Berger et al. (2016) in training a word2vec skip-gram model (Mikolov et al., 2013) using full text citation contexts in S2ORC, where each inline citation span is replaced with its linked paper identifier. When training over this modified text, the word2vec model learns embeddings corresponding to each unique paper identifier, which can be leveraged as paper embeddings. The resulting embeddings shown in Figure 3 and Table 7 form clusters corresponding closely to arXiv Machine Learning categories. Upon inspection, papers of different categories in the same embedding sub-region share research themes (see Table 7), indicating that these paper embeddings trained from citation contexts capture coherent topic similarity and relatedness. These paper embeddings can be used to identify similar papers, using the similarity between two papers' citing contexts as a proxy for paper similarity. The LATEX subset of S2ORC also provides unique opportunities for research. In addition to citations and references, we also extract and parse tables from LATEX source into a structured format. There is an opportunity to use these tables for corpus-level results extraction and aggregation. The LATEX subset also has fine-grained extraction and labeling of mathematical formulas, which can be used to understand proof construction, or to assist in symbol co-reference resolution. The ACL Anthology Network (AAN) (Radev et al., 2009) is a bibliometric-enhanced corpus covering papers in the field of computational linguistics. It is built from the ACL Anthology (Bird et al., 2008) and consists of 24.6k papers manually augmented with citation information. The PubMed Central Open Access corpus is a large corpus of 2.6M papers in the biomedical domain with citations linked to PubMed identifiers. 17 CiteSeerX (Giles et al., 1998), consists of papers collected primarily via web crawl, without integrating metadata provided by sources outside of the PDF. Although citation contexts are no longer available through CiteSeerX, the RefSeer dataset (Huang et al., 2015) 18 is a dataset of short citation context snippets derived from 1.0M papers from CiteSeerX. More recently, Saier and Farber (2019) introduce a corpus built using 1.0M arXiv publications. They use LATEX source to extract text, citation spans and bibliography entries, which are linked to papers in the Microsoft Academic Graph. The citation context they provide are extracted snippets and no bibliography parses are provided. An updated version of this dataset (Saier and Farber, 2020) released concurrently with this work now includes full text. Compared with these resources, S2ORC represents a significantly larger dataset of linked papers covering broad domains of science by leveraging PDF parsing in addition to LATEX source. S2ORC also provides clean full text for text mining and NLP needs with additional enhancements such as annotations of table and figure references and captions. S2ORC's wealth of metadata and structured text allows it to be flexibly adapted to a variety of downstream tasks. We introduce S2ORC, the largest publicly-available corpus of English-language academic papers covering dozens of academic disciplines. S2ORC consists of 81.1M papers, 380.5M resolved citation links, and structured full text from 8.1M open-access PDFs and 1.5M LATEX source files. We aggregate metadata and abstracts from hundreds of trusted sources. Full text is augmented with sections, citation mentions, and references to tables and figures. We demonstrate that S2ORC can be used effectively for downstream NLP tasks in academic paper analysis. The pipeline for creating S2ORC was used to construct the CORD-19 corpus (Wang et al., 2020), which saw fervent adoption as the canonical resource for COVID-19 text mining. CORD-19 is aimed at assisting biomedical experts and policy makers process large amounts of COVID-19 literature in the search for effective treatments and management policies. With over 75K dataset downloads, dozens of search and question-answering systems, and hundreds of participating teams across two shared tasks 19 in the first month of its release, there is little doubt of the resource's impact. Our hope with the release of S2ORC is to ensure such text mining resources are available to researchers even beyond periods of global crisis. We thank Doug Downey, Oren Etzioni, Andrew Head, and Bryan Newbold for their valuable feedback on the manuscript. We also thank Isabel Ca-chola, Dallas Card, Mike D'Arcy, Suchin Guru-rangan, Daniel King, Rik Koncel-Kedziorski, Susan Liu, Kelvin Luu, Noah Smith, Gabi Stanovsky, and Dave Wadden for feedback on the dataset during early development. Finally, we thank the Semantic Scholar team for assisting with data access and system infrastructure.
abstain abstain abstain method abstain abstain abstain abstain abstain other other abstain other abstain other abstain abstain abstain other method abstain method abstain abstain abstain abstain method method method abstain other abstain method abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method method method abstain abstain abstain method abstain abstain abstain method method other abstain abstain abstain abstain abstain other method abstain abstain other abstain method abstain abstain other method method other other other method abstain method method other abstain other method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain other abstain method abstain method abstain abstain other method method other method abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain other abstain abstain abstain method abstain abstain abstain abstain abstain abstain method method abstain method abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain other abstain other abstain abstain abstain abstain abstain method abstain other abstain abstain abstain method abstain abstain other other other other other other other other method other other other abstain abstain abstain abstain objective abstain abstain abstain abstain other other other