The COVID-19 pandemic has spawned a diverse body of scientific literature that is challenging to navigate, stimulating interest in automated tools to help find useful knowledge. We pursue the construction of a knowledge base (KB) of mechanisms a fundamental concept across the sciences, which encompasses activities, functions and causal relations, ranging from cellular processes to economic impacts. We extract this information from the natural language of scientific papers by developing a broad, unified schema that strikes a balance between relevance and breadth. We annotate a dataset of mechanisms with our schema and train a model to extract mechanism relations from papers. Our experiments demonstrate the utility of our KB in supporting interdisciplinary scientific search over COVID-19 literature, outperforming the prominent PubMed search in a study with clinical experts. Our search engine, dataset and code are publicly available. 1 1 Introduction Some experts are familiar with one field, such as AI or nanotechnology [...] no one is capable of connecting the dots and seeing how breakthroughs in AI might impact nanotechnology, or vice versa. Yuval Noah Harari, Homo Deus, 2016 The effort to mitigate the COVID-19 pandemic is an interdisciplinary endeavor the world has rarely seen (Apuzzo and Kirkpatrick, 2020). As one recent example, expertise in virology, physics, epidemiology and engineering enabled a group of 200 scientists to understand and bring attention to the  * Equal contribution. 1 https://covidmechanisms.apps.allenai.org/  a deep learning framework for design of antiviral candidate drugs Temperature increase can facilitate the destruction of SARS-COV-2 gpl16 antiserum blocks binding of virions to cellular receptors ...food price inflation is an unintended consequence of COVID-19 containment measures Retrieved from CORD-19 papers Ent1: deep learning Ent2: drugs Query mechanism relations Ent1: heat Ent2: SARS-CoV-2 Ent1: ? airborne transmissibility of the SARS-CoV-2 virus (Morawska et al., 2020). The diverse and rapidly expanding body of past and present findings related to COVID-19 (Wang et al., 2020b) makes it challenging to keep up, hindering scientists' pace in making new discoveries and connections. Research in natural language processing (NLP) has provided important resources to extract fine-grained relations from scientific papers in specific areas, such as certain subfields of biomedicine (Kim et al., 2013; Nye et al., 2018) or computer science (Wadden et al., 2019). However, these cover only a fraction of all concepts in the literature; in biomedicine alone, there are myriad concepts (Salvadores et al., 2013) not covered by NLP resources. For COVID-19 research, the challenge is especially pronounced due to diversity and emerging concepts; even reading just one paper may require background knowledge in multiple biomedical subfields, physics, chemistry, engineering, computer science and the social sciences. For example, consider a paper studying the indoor dynamics of aerosolized SARS-CoV-2 and the effect of ventilation on transmission by using simulation models, or work on economic impacts of COVID-19 on prices and consumption. To make progress in consolidating such diverse information, we introduce a unified schema of mechanisms as a unified language covering activities, functions and influences across the sciences. These can be proteins that block viral binding, algorithms to design drugs, the effect heat has on viruses, or COVID-19 has on food prices (Fig. 1). We build on the fact that mechanisms underlie much of the natural language of scientific papers (Rhl, 2012), and construct a unified schema with two coarse-grained mechanism relations:  Direct Mechanisms : mechanistic activities (e.g., viral binding) or functions engendered by natural or artificial entities (e.g., a protein used for binding or algorithm used for diagnosis). Indirect Mechanisms : influences and associations such as economic effects of COVID-19 or complications associated with medical procedures. Our coarse-grained relation schema, over freeform text spans, strikes a balance between the granular information extracted by Closed-IE approaches (Freitag, 1998; Hoffmann et al., 2010) and the schema-free breadth of Open IE approaches (Et-zioni et al., 2008; Stanovsky et al., 2018), which often lead to generic and uninformative relations for scientific applications (Kruiper et al., 2020). Furthermore, our schema facilitates construction of a high-quality KB that synthesizes interdisciplinary knowledge. We construct precisely this, releasing MECHANIC ( Mech anisms AN otated in C OVID-19 papers)  an annotated dataset of 2,400 mechanisms based on our schema. We train a state-of-the-art model to extract this information from scientific papers, and use it to build COMB ( C OVID-19 O pen M echanism Knowledge B ase)  a broad-coverage KB of 1.5M mechanisms in COVID-19 papers. We analyze the characteristics of COMB , showing the distribution of relations across scientific subfields and comparing their quality to other IE approaches. We demonstrate the utility of COMB in two studies with experts. In the first study, our system achieves high precision and recall in scientific search with structured queries on both diverse viral mechanisms and applications of AI in the literature. In the second study, we evaluate COMB in a usability study with MDs active in treating and researching COVID-19. Our system is rated higher than PubMed search by the clinical experts, in terms of utility and quality. Our main contributions include:  We introduce a unified schema for mechanisms that generalizes across many types of activities, functions and influences. We construct and distribute MECHANIC , an annotated dataset of papers related to COVID-19, with 2,400 instances of our mechanism relation. Using MECHANIC , we train an IE model and apply it to 160K abstracts in COVID-19 literature, constructing COMB , a KB of 1.5M mechanism instances. Manual evaluation of relations sampled from our KB shows them to have 88% accuracy. We also find a model trained on our data reaches roughly 80% accuracy on a sample of general biomedical papers from across the PubMed corpus, with no additional training, demonstrating the generalization of our approach. We showcase the utility of COMB in structured search for mechanisms in the literature. In a study with MDs working to combat COVID-19, our system is rated higher than PubMed search in terms of utility and quality. Mechanisms in science The concept of mechanisms , also referred to as functional relations , is fundamental across the sciences. For example mechanisms are described in biomedical ontologies (Burek et al., 2006; Rhl, 2012; Keeling et al., 2019), engineering (Hirtz et al., 2002), and across science. Mechanisms can be natural (e.g., the mechanism by which amylase in saliva breaks down starch into sugar), artificial (electronic de-vices), non-physical constructs (algorithms, economic policies), and very often a blend (a pacemaker regulating the beating of a heart through electricity and AI algorithms). Although seemingly intuitive, exact definitions of mechanisms are subject to debate in the philosophy of science (Rhl, 2012; Keeling et al., 2019). An Oxford dictionary definition of mechanisms refers to a natural or established process by which something takes place or is brought about . More intricate definitions discuss complex systems producing a behavior, entities and activities productive of regular changes, a structure performing a function in virtue of its parts and operations, or the Schema Entity types Relations Example SciERC CS methods/tasks (free-form spans) used-for Use GNNs for relation extraction. distinction between correlative property changes and activity determining how a correlative change is achieved (Rhl, 2012). Abstract definitions can help with generalization across many important types of mechanisms. The schema we propose (Sec. 3) is inspired by such definitions, operationalizing them and making them more concrete, and also simple enough for models and human annotators to identify. Information extraction from scientific texts There is a large body of literature on extracting information from scientific papers, primarily in the biomedical sphere. This information often corresponds to very specific types of mechanisms, as shown in Tab. 1. Examples include ChemProt (Li et al., 2016) with mechanisms of chemical-protein regulation, drug interactions in the DDI dataset (Segura Bedmar et al., 2013), genetic and cellular activities/functions in GENIA (Kim et al., 2013), semantic roles of clinical entities (Kilicoglu et al., 2011), PICO interventions and outcomes (Wallace et al., 2016; Nye et al., 2018), and computer science methods/tasks in SciERC (Luan et al., 2018). Such schemas have been used, for example, to extract genomic KBs (Poon et al., 2014) and automate systematic reviews (Nye et al., 2020). Our schema draws on these approaches, but with a much broader reach across concepts seen in COVID-19 papers (Tab. 1, Fig. 2). An important area in information extraction focuses on open concepts, with prominent approaches being Open IE (Etzioni et al., 2008) and Semantic Role Labeling (SRL; Carreras and Mrquez, 2005), which share similar properties and predictions (Stanovsky et al., 2018). While such methods are intended to be domain independent, they perform significantly worse in the scientific domain (Groth et al., 2018). Kruiper et al. (2020) developed a multi-stage process to post-process Open IE outputs, involving trained models and humans to find a balance between generic and fine-grained clusters of relation arguments and omitting noisy clusters. In contrast, our unified schema enables annotating a dataset of mechanism relations between free-form spans and training IE models to automatically generalize across diverse relation types. Our schema is also related broadly to the task of training reading comprehension models on procedural texts describing scientific processes (such as short paragraphs written by crowd workers to explain photosynthesis in simple language; Dalvi et al., 2018). Our representation of scientific texts in terms of a graph of causal relations can potentially help infer processes across science. COVID-19 IE Recent work (Verspoor et al., 2020a) has focused on extracting information from the CORD-19 corpus (Wang et al., 2020b). PICO concepts are extracted and visualized in an exploratory interface in the COVID-SEE system (Ver-spoor et al., 2020b). In Wang et al. (2020a), genes, diseases, chemicals and organisms are extracted and linked to existing biomedical KBs with information such as gene-disease relations. Additional relations based on the GENIA schema are extracted from the text. To address the novel COVID-19 domain, the schema is enriched with new entity types such as viral proteins and immune responses. In this paper, we focus on a more general schema that captures diverse concepts appearing in literature related to COVID-19, an emerging domain with novel concepts coming from many fields and subfields. The mechanism KG we construct includesas a subset diverse biomolecular and clinical information (such as chemical-disease relations) as part of a general mechanism schema. We present a schema that builds upon and consolidates many of the types of mechanisms discussed in Sec. 2. Our defined schema has three key properties: (1) it uses a generalized concept of mechanism relations, capturing specific types of mechanisms in existing schema and extending them broadly; (2) it includes flexible, generic entities not limited to predefined types, and (3) it is simple enough for human annotators and models to identify in the natural language of scientific texts. This schema enables forming our KB by identifying a set of mechanism relations in a corpus of scientific documents (Sec. 4.3). We formally define each mechanism as a relation ( E 1 , E 2 , class ) between entities E 1 and E 2 , where each entity E is a text span and the class indicates the type of the mechanism relation. Entities all share a single common type and can be either natural (e.g., protein functions, viral mechanistic activities) or artificial (e.g., algorithms, de-vices), to capture the generality of the concepts in science (see Fig. 2). We allow each entity to take part in multiple relations (tuples) within a given text, leading to a mechanism graph. Mechanisms are categorized into two coarse-grained classes: 2 Direct mechanisms include activities of a mechanistic nature  actions explicitly performed by an entity, such as descriptions of a virus binding to a cell, and explicit references to a function (e.g., a use of a drug for treatment, or the use of AI for drug design as in Fig. 1). Indirect mechanisms include influences or associations without explicit mechanistic information or mention of a function (such as describing observed effects, without the process involved). These relations correspond more to input-output cor-2 We also provide a dataset and extraction model for ternary relations in the form of (subject, object, predicate) . We focus on the coarse-grained mechanism schema due its broader flexibility and coverage. See App. A.1 for details. CS / m a t h / e n g . b i o m e d m e t h o d s c h e m i s t r y / p h y s i c s e c o l o g y / z o o l o g y e p i d e m i o l o g y g e n e t i c s i m m u n o l o g y m e d ./ p h a r m a m o l e c . b i o . s o c i a l / p u b li c v i r o l o g y + m i c r o b i o . 0 50 100 150 200 250 Figure 2: MECHANIC covers a diverse set of scientific fields. Histogram of domains in MECHANIC (sample of 350 relations). Manually labeled relation entities, based on a list of scientific disciplines from Wikipedia. relations (Rhl, 2012), such as indicating that COVID-19 may lead to economic impacts but not how (Fig. 1), as opposed to direct mechanisms describing inner workings  revealing more of the intermediate states that lead from initial conditions (COVID-19) to final states (price inflation) or explicitly describing a function. As an example for the utility of this distinction between direct and indirect relations, consider an MD looking to generate a structured list of all uses of a treatment (direct mechanism), but not include side effects or complications (indirect). We describe our approach (depicted in Fig. 3) for extracting a knowledge base of mechanisms using our unified schema. We first curate MECHANIC , an annotated dataset of general mechanisms from a small collection of scientific papers (Sec. 4.1). We then train a model on our annotated data to extract mechanism relations from the entire CORD-19 corpus of scientific papers; we use it to build COMB , a knowledge base of mechanisms across the entire CORD-19 corpus of (Sec. 4.2), which supports semantic search for relations (Sec. 4.3). We construct a dataset of mechanism relations in texts randomly sampled from the CORD-19 corpus (Wang et al., 2020b) that includes scientific papers connected to COVID-19. To circumvent annotation challenges in scientific datasets (Luan et al., 2018) and ensure high-quality annotations, we follow a three-stage process of (1) annotating entities and relations using biomedical experts, (2) unifying span boundaries with an NLP expert, and (3) verifying annotations with a bio-NLP expert. Our annotation process is a relatively low-resource and generalizable approach for a rapid response to the ...deep reinforcement learning can be used to learn mitigation policies in epidemiological models... In the first stage, five annotators with biomedical and engineering background annotate all mechanism relations as defined in Sec. 3 (full annotation guidelines are available in our code repository). Relations are annotated as either direct/indirect. Entities are annotated as the longest span of text that is involved in a relation with another entity, while not including redundant or irrelevant tokens. As in related tasks (Luan et al., 2018), annotators are guided to resolve doubt on span boundaries by selecting the longest relevant span. Annotators had a one-hour training session. In the first part of the training session, annotation guidelines were reviewed. The guidelines included simple explanations of direct/indirect mechanisms along with introductory examples (e.g.,  the virus makes use of spike protein to bind to a cell ,  A virus leads to respiratory infection ). In the second part, annotators saw examples from papers in the annotation interface (see Fig. 6, App. A), and performed a few live training annotations. We initially observed significant variation between annotators in identifying span boundaries for entity annotations, stemming from inherent subjectivity in such annotation tasks (Stanovsky et al., 2018; Luan et al., 2018) and from lack of NLP experience by some annotators. In the second stage, an NLP expert annotator conducted a round of style unification by viewing annotations and adjusting span boundaries to be more cohesive while preserving the original meaning, focusing on boundaries that capture essential but not redundant or generic information (e.g., adjusting the span substantial virus replication by unknown mechanisms to include only virus replication ). Finally, in the third stage, a bio-NLP expert with experience in annotating scientific papers verified the annotations and corrected them as needed. The expert accepted 81% of the annotations from the second stage without modification, confirming the high quality of the stage-2 data. Relation label mismatches accounted for 5% of the remaining 19%. Other sources of disagreement were span mismatches and new relations added by the bio-NLP expert adjudicator. The resulting dataset (MECHANIC : Mech anisms AN otated in C OVID-19 papers) contains 2,370 relation instances (1645 direct, 725 indirect) appearing in 1,000 sentences from 250 abstracts. 3 Average span length is 4 tokens, while the average distance between relation arguments is 11.40 tokens. Using MECHANIC , we train an IE model to extract mechanism relations from sentences in scientific documents. We train DyGIE++ (Wadden et al., 2019), a state-of-the-art end-to-end IE model which extracts entities and relations jointly (without assuming to have entity spans given), classifying each relation as one of { DIRECT , INDIRECT } . 4 To form our corpus-level KB, we apply the trained model to each document in our corpus (all 160K abstracts in the CORD-19 corpus) to extract mechanism relations and then integrate the extracted relations. We find that our trained model achieves high precision scores for high confidence predictions (precision  80% within top20 predicted relations; see P @ K figure, App. B). Therefore, our corpus-level KB is constructed by filtering predictions with low confidence. 3 The dataset is similar in size to related scientific IE datasets (Luan et al., 2018) which share related challenges in collecting expert annotations of complex or ambiguous concepts over difficult texts. 4 We use DyGIE++ with SciBERT (Beltagy et al., 2019) embeddings fine-tuned on our task and perform hyperparameter grid search (for dropout and learning rate only) and select the best-performing model on the development set ( 7 e  4 and 0 . 43 , respectively). Full details are in App. B.3. To integrate relations and entities across the corpus, we use standard surface-level string normalization (such as removing punctuation, lemmatiz-ing, and lowercasing) and unify and normalize entity mentions using coreference clusters of entities within a document. 5 Each coreference cluster is assigned a representative entity as the mention with the longest span of text, and all other entities in that cluster are replaced with the representative entity. This is particularly useful for normalizing pronouns such as it with the original mention they referred to (e.g., a specific virus or method it refers to). Our final KB (COMB ) consists of 1.5M relations in the form of ( E 1 , E 2 , DIRECT/INDIRECT ) fil-tered by high confidence score ( >= 90% ), where entities E i are standardized free-form spans of text. The constructed KB enables applications for retrieving relations across concepts from many disciplines. For example, searching for all documents that include mechanisms to incorporate AI in studies of heart disease ( E 1 = AI , E 2 = heart disease , DIRECT ) requires going beyond simply finding documents that mention AI and heart disease . Here, we describe our approach for searching over the KB by encoding entities and relations, capturing related concepts (such as cardiac disease and heart conditions ), as well as simpler surface matches ( artificial intelligence methods , artificial intelligence models ). Specifically, for a given query q  ( E q 1 , E q 2 , class ) , our goal is to find mechanisms r i in COMB whose entities are free-form texts similar to E q 1 , E q 2 in the query. The class is used to filter for the type of relationfor example, when explicitly requiring DIRECT mechanisms. Entity encoding We obtain an encoding function f  E  R d to encode all unique spans (entities) in the KB to a d dimensional vector space. The encoding function is derived by fine-tuning a language model (LM) originally trained on PubMed papers (Gururangan et al., 2020) on semantic similarity tasks. For fine-tuning, we use sentence pairs in STS (Cer et al., 2017) and SNLI (Bowman et al., 2015) following Reimers and Gurevych (2019), and add biomedical sentence pairs from the BIOSSES dataset (Sogancoglu et al., 2017). 5 We use a pre-trained DyGIE++ model trained on SciERC to obtain coreference clusters. Relation similarity Given a query q , we rank the set of all COMB relations with the same class as the query. For each candidate relation r = ( E 1 , E 2 , class ) in COMB , we compute its similarity to the query relation q as the minimum similarity between encodings of their corresponding entities: min j  { 1 , 2 } f ( E j )  f ( E qj ) . With this definition, a relation ( E 1 , E 2 ) with E 1 very similar to the first entity of the query E q 1 but E 2 distant from E q 2 will be ranked low. For example, with the query ( E q 1 = deep learning , E q 2 = drugs ) , the relation ( E 1 = microscope , E 2 = drugs ) will be ranked low due to the pair (deep learning, microscope). For efficient search, we create an index of embeddings corresponding to the 900K unique surface forms in COMB and employ a system designed for fast similarity-based search (Johnson et al., 2017). In this section, we evaluate the constructed KB of mechanisms in terms of correctness and informativeness (Sec. 5.1), and its utility in searching for mechanisms (Sec. 5.2). Our main goal is to ensure the mechanism relations have high quality to support our large-scale KB and search applications. We further show that our schema is useful as compared to other schema. We employ two annotators with biomedical and CS backgrounds to judge the quality of the predicted relations in COMB . In particular, following Groth et al. (2018), annotators are given a predicted relation together with the sentence from which it was extracted. We collapse all entities/relations into one generic type for this analysis. Annotators are asked to label the predicted relation as correct if (1) it accurately reflects a mechanistic relation mentioned in the sentence ( correctness ), and (2) the extracted entities and relation label are sufficient to convey the meaning of the relation, without referring to the source sentence ( informativeness ). We collect human judgements for 300 predicted relations for our approach and baselines, sampled from 150 randomly selected sentences. Agreement is 71% by Cohen's Kappa and 73% by Matthew's Correlation Coefficient. Comparing KB quality to other schemas To showcase the benefit of our approach, we compare the relations extracted using a DyGIE model trained on MECHANIC , versus a DyGIE model MECHANICS c i ERCSRLS e m R ep O pen IE 10 20 30 40 50 60 70 80 90 A cc u r a cy AI Viral Metric PubMed COMB Search 71% 90% Utility 69.5% 92% Interface 78% 90% Overall 74% 91% Figure 4: Evaluating COMB in studies with experts. trained on other resources that are most related to our mechanisms: SemRep (Kilicoglu et al., 2011) captures a wide range of biomedical relations (such as drug-drug interactions), and SciERC (Luan et al., 2018) contains relations relevant to computer science (such as method-task and used-for rela-tions). 6 In addition, we compare with a Semantic Role Labeling (SRL) method (Shi and Lin, 2019) that captures broad relations between free-form spans that focus on agents and actions, and a neural OpenIE model (Stanovsky et al., 2018). Fig. 4 (left) shows that 88% of relations from COMB are marked as correct by human raters, demonstrating that our approach extracts mechanism relations with better quality than external resources. 7 These results suggest that our predicted relations are of overall high quality and can be used to build our corpus-level KB and explore its utility. Examining Generalization COVID-19 papers are highly diverse both topically and chronologically. We conduct a small-scale preliminary experiment examining whether a model trained on MECHANIC can generalize to capture mechanism relations in the general biomedical papers, from a much larger corpus of open access papers on PubMed Cen-6 We use an expert annotator to align external resources to our direct or indirect mechanism annotations, e.g., USED-FOR is mapped to direct mechanism). 7 We also experiment with automated evaluation. We split MECHANIC into train/dev/test sets (170/30/50 abstracts), and obtain F 1 = 50 . 2 for entity detection, F 1 = 45 . 6 for relation detection and F 1 = 42 . 8 for classification, on par with performance in other similar scientific IE tasks (Luan et al., 2018). See more details in App. B.4. tral (PMC). 8 We randomly sample a set of 200 predicted relations from papers across the entire PMC corpus, and label them using the same criteria used above. As expected, we find that performance drops, but encouragingly is still considerably high: after filtering predictions with confidence lower than 90% in the same way we construct COMB , 76% of relations are considered correct. When filtering for confidence with a threshold of 95% (which captures 70% of the samples), the rate of correct predictions is 78%. In future work it would be interesting to fine-tune our model on a small set of labeled examples from the general PMC corpus to potentially improve these results. We design several search tasks and user studies to evaluate the utility of the constructed KB (Sec. 5.2.1) and compare it with the PubMed medical KB and search engine (Sec. 5.2.2), as judged by medical doctors working on the front lines of COVID-19 treatment and research. All tasks are designed to evaluate our framework's utility in helping researchers and clinicians looking to quickly search for mechanisms or cause-effect relations in the literature and retrieve a list of structured results. We form search queries based on a wide range of topics pertaining to (1) SARS-CoV-2 mechanisms (such as modes of transmission, drug effects, climatic influences, molecular-level properties) and (2) applications of AI in this area. Tab. 2a and 2b show queries and example relations returned from COMB , along with the context sentences from which they were extracted. Viral mechanism search Queries are formed based on statements in recent scientific claim-verification work (Wadden et al., 2020; see full list in App. C.2). For example, for the statement the coronavirus cannot thrive in warmer climates , we form the query as ( E 1 = Warm climate , E 2 = coronavirus ) (see Tab. 2a row 1). For statements reflecting an indirect association/influence, we filter for INDIRECT relations (Tab. 2a row 2). For statements that reflect an undirected mechanism relation (e.g., Lymphopenia is associated with severe COVID-19 disease ), we query for both directions. AI applications search This task is designed to explore the uses of AI in COVID-19 papers (Tab. 2b). We use queries where the first entity E 1 is a leading subfield or method within AI (e.g., deep reinforcement learning or text analysis ), and the second entity E 2 is left unspecified. Since all queries relate to uses of AI, we filter for DIRECT relations. These open-ended queries simulate an exploratory search scenario, and can potentially surface inspirations for new applications of AI against COVID-19 or help users discover where AI is being harnessed. if the sentence actually expresses the mechanism. These annotations are used as ground-truth labels to compute precision/recall scores of the relations extracted by our algorithm. Since it is not feasible to label every relation, annotators are shown a list of 20 relations for each query including high and low rank relations returned by our search algorithm. 9 In total, we use 5 annotators to obtain 1,700 relevance labels across both tasks. Inter-annotator agreement is high by several metrics, ranging from 0 . 7  0 . 8 depending on the metric and task; see App. C.2. Annotators have graduate/PhD-level background in medicine or biology (for the first task) and CS or biology (for the second task). Results Fig. 4 (center) shows our results for both tasks. For biomedical search queries, we observe 90% precision that remains stable for recall values as high as 70% . For AI applications we observe a precision of 85% at a recall of 40% that drops more quickly. This lower precision is likely due to the fact that E 2 is unspecified, leading to a wider range of results with more variable quality. be-9 Specifically, for each query we retrieve the top-1000 similar relations from COMB , ranked as described in Sec. 4, and select the top and bottom 10 relations (20 per query, 200(=20x10) per task, 400(=200x2) in total), shuffle their order, and present to annotators together with the original sentence from which each relation was extracted. This experiment compares the utility of COMB in structured search for causal relationships of clinical relevance to COVID-19 with PubMed 10 a prominent search engine for biomedical literature that clinicians and researchers frequently peruse as their go-to tool. PubMed allows users to control structure (e.g., with MeSH terms or pharmacological actions), is supported by a KB of biomedical entities used for automatic query expansion, and has many other functions. Expert evaluation We recruit five expert MDs with a wide range of specialities including gastroenterology, cardiology, pulmonary and critical care who are active in treating COVID-19 patients and in research. Each expert completed search randomly ordered tasks using both PubMed and our COMBUI, showing the full set of ranked relations, as well as the sentence snippet mentioning the relation, the paper title, and hyperlink to abstract. At the end of the study after all search tasks are completed for both our system and PubMed, experts are given a questionnaire of 21 7-point Likert-scale questions to judge system utility, interface, and search quality. The first 16 questions are taken from a Post Study System Usability Questionnaire (PSSUQ; Lewis, 2002) widely used in system quality research. The last 5 questions are designed by the authors to evaluate search quality such as overall result relevance and ranking (for the full question list, see App. C.2). Each question is asked twice, once for PubMed and once for our system, leading to 21  2  5 = 210 responses. Search queries We provide experts with seven search queries that were created by an expert medical researcher, relating to causal links (e.g., between COVID-19 and cardiac arrhythmias) and functions (e.g., Ivermectin as a treatment). See full set of queries in App. C. Results Fig. 4 (right) shows the average Likert scores (normalized to [0%,100%]) across all questions and users for COMB and PubMed. The results show that the medical experts strongly prefer COMB to PubMed (overall average of 91% vs. 74%, with non-normalized scores of 6.6 vs. 5.2). On average across the 21 questions, the majority of the five experts assigned our interface a higher score than PubMed, at an average rate of 3.5/5. This rate in-10 https://pubmed.ncbi.nlm.nih.gov/ creases further when considering tieson average 4.75/5 of the experts assigned our system a score equal or higher than PubMed. Overall, our system significantly outperforms PubMed in this task, with an average gap of roughly 20% for search and utility-related questions (Wilcoxon signed rank test p-value is significant at 4 . 77  10  7 ). These results are particularly interesting and indicate the potential of COMB because of the experts' strong familiarity with PubMed and the simple nature of our UI. Our system searches and retrieves relations  only texts explicitly mentioning relations that match the input query. This often more precisely reflects the query than results returned by PubMed, which do not have the additional layer of structured information in COMB . For example, for the query ( E 1 =cardiac arrhythmias, E 2 =COVID-19), PubMed returns the following title of one paper: Guidance for cardiac electrophysiology during the COVID-19 pandemic [....] Electrocardiography and Arrhythmias Committee  E 1 and E 2 are both mentioned, but not within a mechanism relation. We introduced a unified schema for mechanisms that generalizes across many types of activities, functions and influences. We constructed and distributed MECHANIC , a dataset of papers related to COVID-19 annotated with this schema. We trained an IE model and applied it to COVID-19 literature, constructing COMB , a KB of 1.5M mechanisms. We showcased the utility of COMB in structured search for mechanism relations in COVID-19 literature. In a study with MDs active in the fight against the disease, our system is rated higher than PubMed search for both utility and quality. Our unified view of mechanisms can help generalize and scale the study of COVID-19 and related areas. More broadly, we envision a KB of mechanisms that enables the transfer of ideas across the literature (Hope et al., 2017), such as by finding relationships between mechanisms in SARS-CoV-2 and other viruses, and assists in literature-based discovery (Swanson and Smalheiser, 1996) by finding cross-document causal links. Our knowledge-base and search system is primarily intended to be used by biomedical researchers working on COVID-19, and researchers from more general areas across science. Models trained and developed on our dataset are likely to serve researchers working on COVID-19 information extraction, and scientific NLP more broadly. We hope our system will be helpful for accelerating the pace of scientific discovery, in the race against COVID-19 and beyond. Our knowledge-base can include incorrect information to the extent that scientific papers can have wrong information. Our KB includes metadata on the original paper from which the information was extracted, such as journal/venue and URL. Our KB can also miss information included in some papers. Our data collection process respected intellectual property, using abstracts from CORD-19 (Wang et al., 2020b), an open collection of COVID-19 papers. Our knowledge-base fully attributes all information to the original papers. All annotators were given extensive background on our objectives, and told their annotations will help build and evaluate a knowledge-base and search engine over COVID-19 research. Graduate-student annotators were payed 25 USD per hour. MD experts helped evaluate the tool on a voluntary basis. We like to acknowledge a grant from ONR N00014-18-1-2826. Authors would also like to thank anonymous reviewers, members of AI2, UW-NLP and the H2Lab at The University of Washington for their valuable feedback and comments.
abstain abstain objective method objective abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain method objective method abstain method method result objective objective method abstain objective method method result objective abstain abstain other other other other other other other other objective other other other other abstain other other other objective abstain abstain other other other other other objective method method abstain method method abstain method abstain abstain other abstain method objective method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method result method abstain method abstain abstain method abstain abstain abstain abstain abstain method objective abstain result abstain method result abstain method abstain abstain method method abstain abstain method abstain abstain abstain method abstain method method method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain method abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain other abstain abstain method abstain abstain abstain method method abstain abstain abstain result method abstain abstain abstain abstain method abstain method abstain abstain abstain abstain other other