Abstract Intrinsic evaluations of OIE systems are carried out either manuallywith human evaluators judging the correctness of extractions or automatically, on standardized benchmarks. The latter, while much more cost-effective, is less reliable, primarily because of the incompleteness of the existing OIE benchmarks: the ground truth extractions do not include all acceptable variants of the same fact, leading to unreliable assessment of the models' performance. Moreover, the existing OIE benchmarks are available for English only. In this work, we introduce BenchIE: a benchmark and evaluation framework for comprehensive evaluation of OIE systems for English, Chinese, and German. In contrast to existing OIE benchmarks, BenchIE is fact-based , i.e., it takes into account informational equivalence of extractions: our gold standard consists of fact synsets , clusters in which we exhaustively list all acceptable surface forms of the same fact. Moreover, having in mind common downstream applications for OIE, we make BenchIE multi-faceted ; i.e., we create benchmark variants that focus on different facets of OIE evaluation, e.g., compactness or minimality of extractions. We benchmark several state-of-the-art OIE systems using BenchIE and demonstrate that these systems are significantly less effective than indicated by existing OIE benchmarks. We make BenchIE (data and evaluation code) publicly available. 1 1 Introduction Open Information Extraction (OIE) is the task of extracting relations and their arguments from natural language text in a schema-free manner (Banko et al., 2007). Consider the sentence "Sen. Mitchell, who is from Maine, is a lawyer." ; an OIE system is expected to extract the triples ("Sen. Mitchell"; "is from"; "Maine") and ("Sen. Mitchell"; "is"; "a lawyer") from the sentence. OIE systems are used 1 https://github.com/gkiril/benchie in many downstream tasks, including knowledge graph (KG) population (Gashteovski et al., 2020), open link prediction (Broscheit et al., 2020), and question answering (Yan et al., 2018). These downstream tasks lend themselves as natural setups for extrinsic OIE evaluation (Mausam, 2016). While valuable in concrete applications, such extrinsic evaluations do not measure the intrinsic correctness of the extracted facts: for that purpose, several benchmarks for intrinsic OIE evaluation have been proposed (Stanovsky and Dagan, 2016; Lechelle et al., 2019; Bhardwaj et al., 2019). Automated benchmark evaluations are more feasible (i.e., faster and cheaper) than manual OIE evaluations (Hohenecker et al., 2020). The current benchmarks, however, use scoring functions that are based on approximate (token-level) matching of system extractions against ground truth facts, which seems to be substantially less reliable than human judgments of extraction correctness (Zhan and Zhao, 2020). This primarily stems from the incompleteness of existing OIE benchmarks: the gold standard extractions do not include all acceptable surface realizations of the same fact . Consider, for example, a sentence from the recent evaluation framework CaRB (Bhardwaj et al., 2019): Sen. Mitchell is confident he has sufficient votes to block such a measure with procedural actions ; with the gold triple extraction ( Sen. Mitchell ; is confident he has ; sufficient votes to . . . procedural actions ). Intuitively, a system extraction with a more concise object ( Sen. Mitchell ; is confident he has ; sufficient votes )could also be accepted, as it still captures the same core piece of knowledge, and would arguably be valuable in most downstream tasks. To account for this, existing benchmarks credit system extractions for per-slot lexical overlap with gold extractions. Such scoring is overly lenient and overestimates the systems' ability to extract correct knowledge facts . Consider, e.g., a system 4472 extraction ( Sen. Mitchell ; is confident he has ; procedural actions ) for the above-mentioned sentence. From the factual perspective, this extraction is clearly incorrect ( Sen. Mitchell has votes , not actions ). However, the popular CaRB benchmark with its token-level metrics would judge the extraction as having (1) perfect precision, since all extracted tokens can be found in corresponding slots of a gold extraction and (2) high recall, as all of the gold subject and predicate tokens as well as two gold object tokens ( procedural and actions ) are found within corresponding slots of the system extraction (Table 1). Moreover, by providing a single ground truth extraction per fact, existing OIE benchmarks fail to acknowledge that different downstream applications focus on different facets (i.e., aspects) of OIE extractions: e.g., for text summarization, one may prefer minimal extractions (Ponza et al., 2018), whereas knowledge base population benefits from strict correctness of entities in subject and object slots (Lin et al., 2020). In this work, we depart from lenient OIE evaluations based on per-slot token overlaps and propose BenchIE, a novel fact-centric and multi-faceted OIE evaluation framework and benchmark at the core of which is the following question: Does the system extraction express the same fact (i.e., the same unit of knowledge) as any of the ground truth extractions (and vice versa) w.r.t. the specific aspect of the OIE extraction that is of interest for one or more downstream applications? Contributions. BenchIE advances the state of the art in OIE evaluation in the following: (1) it is the first fact-centered approach to OIE evaluation: to reliably answer the above question, we exhaustively list all correct extractions of the same fact. In contrast to existing benchmarks, BenchIE specifies complete sets of fact-equivalent extractions (dubbed fact synsets ), allowing us to avoid error-prone evaluation based on token overlap measures; (2) BenchIE is the first multi-faceted OIE benchmark, allowing to test systems for different aspects of OIE extractions that may be relevant in concrete downstream applications; (3) BenchIE is a multilingual benchmark, covering English, Chinese, and German, and to the best of our knowledge the first with manually annotated (i.e., gold standard) extractions in all languages; 2 (4) finally, as a 2 Ro et al. (2020) introduce a multilingual version of the CaRB dataset by machine translating both sentences and extractions. fact-based and multi-faceted benchmark, BenchIE allows us to perform what we believe to be the most comprehensive profiling and comparative evaluation of OIE systems. BenchIE portrays fact extraction abilities of six state-of-the-art OIE models much less favorably and points to their limitations that cannot be detected with existing benchmarks. Most OIE systems extract (subject, predicate, object) triples, with concepts as subjects and objects and verb phrases (VPs) as predicates (Banko et al., 2007; Stanovsky et al., 2018; Lauscher et al., 2019; Gashteovski et al., 2017, 2019), though systems producing n-ary (Akbik and Lser, 2012), nested (Bhutani et al., 2016), and noun-mediated extractions (Yahya et al., 2014) also exist. Here we follow the most common practice and focus on VP-mediated facts. Our novel fact-based benchmark and evaluation paradigm can, however, equally be applied to other types of extractions (e.g., Friedrich et al. (2022) used this fact-based concept for OIE to create gold annotations for NE-Centric OIE triples ; i.e., triples where each argument is a named entity and the relations could be either verb phrases or noun phrases). We introduce the general concept of a fact synset : a set of all possible extractions (i.e., different surface forms) for a given fact type (e.g., VP-mediated facts) that are instances of the same fact. E.g., given the input sentence from Table 2, the extractions ( Sen. Mitchell ; has sufficient votes to block ; such a measure ) and ( Sen. Mitchell ; has sufficient votes to block ; measure ) capture the same fact and thus belong to the same fact synset. Existing benchmarks fail to exhaustively list all acceptable extractions for the same fact. This is precisely why, in order to avoid penalizing systems for correct extractions that are not exactly the same as the gold triples, they resort to lenient token-based performance measures prone to two types of errors: (1) they punish correct fact extractions that have limited lexical overlap with the gold extraction of the same fact, e.g., ( Sen. Mitchell ; is confident he has ; sufficient votes) vs. ( Sen. Mitchell ; is confident he has ; sufficient votes to . . . procedural actions ) unreliable for OIE  as shown by Kotnis et al. (2022), up to 70% of sentence or extraction translations obtained this way were incorrect. and (2) they reward incorrect extractions that have high lexical overlap with a gold extraction, e.g., ( Sen. Mitchell ; is confident he has; procedural actions ) vs. (Sen. Mitchell; is confident he has; sufficient votes to block. . . with procedural actions) . To prevent this, BenchIE relies on exact matching of system extractions against the gold fact synsets. Further, some OIE systems (over)generate extractions of the same fact; e.g., (Sen. Mitchell; has sufficient votes to block; such a measure) and ("Sen. Mitchell"; "has sufficient votes to block"; "measure") . Existing evaluation procedures do not acknowledge the fact equivalence of extractions and consequently reward OIE systems for multiply extracting the same fact. Our evaluation based on fact synsets directly remedies these shortcomings of existing OIE benchmarks. English Benchmark. To make BenchIE comparable to previous benchmarks, we annotate fact synsets on a subset of sentences from CaRB (Bhard-waj et al., 2019). Because exhaustive annotation of fact synsets is time consuming, we carried it on 300 (out of 1,200) randomly sampled CaRB sentences. To collect truly exhaustive fact synsets, two expert annotators independently labeled the selected 300 sentences in three rounds. (1) Each annotator first (independently) manually denoted every extraction in which a VP-predicate connects two concepts. The annotator then grouped the fact-equivalent triples into fact synsets. 3 To speed the annotation process up, we developed a dedicated web-based annotation tool AnnIE that facilitates the extraction of VP-mediated triples (e.g., we color-code verbs to indicate possible predicate heads) and their clustering into fact synsets; 4 (2) The annotators then carefully examined all gold extractions from the original CaRB dataset and added those judged to be correct, yet missing from the manually labeled fact synsets from the previous step; (3) Finally, each annotator compared the extractions of all OIE systems in evaluation (see 4) against the BenchIE's fact synsets (i.e., the result of the first two steps). Any system extraction not found in BenchIE was carefully examined andif judged to be correct added to the appropriate fact synset. 5 Finally, the 3 We provide the annotation guidelines in Appendix A.1. details about the tool, see Friedrich et al. (2022). 5 Very few extractions were actually added in steps (2) and (3); i.e., there were very few correct extractions (from CaRB gold standard and output of OIE systems) that the annotators missed during manual annotation of fact synsets. two annotators merged their independently created annotations by discussing and jointly resolving the disagreements. The overall annotation effort for the English dataset amounted to 80 hours per annotator. English BenchIE contains 136,357 unique gold extractions, grouped into 1,350 fact synsets. For comparison, CaRB (Bhardwaj et al., 2019) lists mere 783 gold triples for the same 300 sentences. Table 2 shows fact synsets for an example sentence. Inter-Annotator Agreement (IAA). To validate BenchIE's annotations, we measure the inter-annotator agreement (IAA) between our two expert annotators. To this end, we quantify the agreement via recall at the fact level (see 2.3 for further details): for each annotator, we compute their fact-level recall as the percentage of fact synsets of the other annotator they cover with their extractions. 6 We average the fact-level recalls of the two annotators as the IAA score. We observed a high IAA score of 0 . 79 . Upon manual inspection, we found that the annotators mostly agree on fact-synset level; most of the the disagreements are on extractions level (particularly, from marking the optional tokens within an extraction; see Appendix A.1.3 for details about the optional tokens). Chinese and German Benchmarks. Two bilingual expert annotators  native in the target language and fluent in English ( EN )  translated the original 300 English sentences to Chinese ( ZH ) and German ( DE ), respectively. Then, to collect exhaustive fact synsets in ZH and DE , they followed the same three annotation rounds described for 2.2. Due to substantial (primarily syntactic) differences compared to EN , we adjusted the annotation guidelines for these languages (see the Appendix A.2 and A.3 for more details). The statistics (number of fact synsets and extractions) of the ZH and DE benchmarks are given in Table 3. Compared to EN BenchIE, the ZH benchmark contains significantly fewer fact synsets (994 compared to 1,350) and more than two orders of magnitude fewer extractions. The drastically smaller number of extractions is primarily due to the lack of determiners and articles in Chinese. Their frequent occurrence in English combined with their neutrality w.r.t. extractions' correctness results in many mutually different yet fact-equivalent extractions. The numbers for German are, expectedly, much closer to those for English. We assume that BenchIE is (1) complete , i.e., that it contains (a) all VP-mediated facts expressed in input sentences and (b) for each fact, its every acceptable extraction as well; and (2) sound , i.e., that it does not contain any incorrect extraction that would capture a fact not stated in the sentence. Such a complete OIE gold standard enables not only a more reliable evaluation of OIE systems by means of exact matching, but also an evaluation at the more meaningful level of knowledge facts, rather than at the level of individual triples. Concretely, we consider a system extraction to be correct if and only if it exactly matches some gold extraction from some fact synset. The number of true positives (TPs) is the number of fact synsets (i.e., different facts) covered by (at least one of the) system extractions. This way, a system that extracts N different triples of the same fact, will be rewarded only once for the correct extraction of the fact. BenchIE's false negatives (FNs) are then, intuitively, fact synsets not covered by any of the system extractions. Finally, each system extraction that does not exactly match any gold triple (from any synset) counts as a false positive (FP). We then compute Precision , Recall , and F 1 score (as the final score) from TP, FP, and FN in standard fashion. Different downstream applications care about different aspects of OIE extractions. For IE-based text summarization and simplification (Ponza et al., 2018; tajner and Glava, 2017), e.g., triples should be minimal overall, across all slots (i.e., without unnecessary tokens), but the exact token placement across the slots (e.g., if a preposition is in the predicate or object) does not matter. For entity linking and knowledge base population (Lin et al., 2020), in contrast, the token placement between slots is critical: a token that is not part of an entity, should not be placed into subject or object. Acknowledging this, we create three additional variants of the English BenchIE, referred to as facets , each 4475 Input sentence: "Sen. Mitchell is confident he has sufficient votes to block such a measure with procedural actions." BenchIE-E ("Sen. Mitchell" | "he"; "is confident he has ... [such] [a] measure with"; "procedural actions") BenchIE-C "(Sen. Mitchell | he) is confident he has sufficient votes to block [such] [a] measure with procedural actions" BenchIE-M ("Sen. Mitchell" | "he"; "is confident he has sufficient votes to block measure with"; "procedural actions") ("Sen. Mitchell" | "he"; "is confident he has sufficient votes to block measure"; "with procedural actions") Table 4: Illustration of BenchIE's facets for one fact synset ( f 4 from Table 2): all acceptable surface realizations under each facet are shown. corresponding to one aspect that is relevant in common OIE applications. This effort addresses recent calls for multi-dimensional analysis of NLP systems (Ethayarajh and Jurafsky, 2020; Narayan et al., 2021) and is well-aligned with recent efforts that create multi-faceted benchmarks for other NLP tasks (Liu et al., 2021; Vth et al., 2021) and datasets (Xiao et al., 2022). The default, general-purpose BenchIE facet from the previous section was designed to be somewhat tolerant to token distribution accross slots (see Appendix A.1.2 for details): some tokens may be placed in either the predicate or object (e.g., the preposition with in the synset f 4 in Table 2). This enables a more flexible comparison of OIE systems that are designed for different purposes (i.e., systems that produce slightly different token placements are not punished) and is in line with prior work on intrinsic OIE evaluation, both automatic (Stanovsky and Dagan, 2016; Bhardwaj et al., 2019) and manual (Fader et al., 2011; Del Corro and Gemulla, 2013; Gashteovski et al., 2017). Such extraction flexibility, however, may not be desirable in tasks like automated KG construction (Wolfe et al., 2017; Jiang et al., 2019) or entity linking (Lin et al., 2020, 2021). Angeli et al. (2015) show empirically that extractions with wholesome entities and without additional tokens yield benefits in KG construction. Since OIE is predominantly used for KG-related tasks (Weikum et al., 2020), it is paramount to have an evaluation facet that imposes strict(er) token boundaries on entity slots  subjects and objects. We thus create the entity facet of the benchmark (BenchIE-E) with this additional constraint of wholesomeness of subject and object concepts. BenchIE-E was constructed by one of our annotators (see 2.2) by removing from EN BenchIE's fact synsets the extractions in which subject and/or object was not a wholesome concept (see Table 4). The default BenchIE facet (2) compares OIE extractions against gold triples from fact synsets at the slot level: to be judged correct, an extraction must exactly match some gold triple in all slots. This criterion, however, is overly strict if extractions are to be used in applications like summarization or simplification (Ponza et al., 2018; tajner and Glava, 2017), which commonly concatenate the content of the slots. In this case, it does not matter if a sequence of tokens occurs at the end of the subject or beginning of the predicate (analogously for predicate and object). To reflect this, we introduce the concatenation facet , BenchIE-C: for each gold BenchIE triple, we create the gold BenchIE-C utterance by simply concatenating the content of the triple's slots (see Table 4). Our third additional evaluation facet addresses the aspect of minimality of OIE extractions (Gash-teovski et al., 2017). More compact extractions can benefit both text generation (Ponza et al., 2018; tajner and Glava, 2017) and KG-related tasks (Lin et al., 2020, 2021). If two triples t 1 and t 2 capture the same fact (i.e., are in the same fact synset), t 1 is considered more compact than t 2 if tokens of each t 1 slot make a (non-strict) subsequence of tokens in the corresponding t 2 slot (Gashteovski, 2020). 7 To allow for evaluation of minimality, BenchIE-M triples contain only the non-optional tokens (denoted in square brackets in Table 2) from the corresponding BenchIE triple. Consequently, BenchIE-M fact synsets on average contain many fewer extractions than the original BenchIE synsets. 8 4 Fact-Level Evaluation We first compare BenchIE's fact-level evaluation (i.e., default facet, 2) against CaRB's token-level 7 At least one t 1 slot has to be a strict subsequence of the respective t 2 slot; t 1 and t 2 would be the same otherwise. scoring (Bhardwaj et al., 2019). 9 Our quantitative results confirm our intuitions and observations (see Table 1): CaRB systematically and substantially overestimates OIE systems' performance. BenchIE, we argue, portrays the fact extraction abilities of OIE systems more realistically. OIE Systems. We tested six widely used OIE systems that extract VP-mediated facts for EN , namely: ClausIE (Del Corro and Gemulla, 2013), Stanford OIE (Angeli et al., 2015), MinIE (Gash-teovski et al., 2017), ROIE (Stanovsky et al., 2018), OpenIE 6 (Kolluru et al., 2020) and M 2 OIE (Ro et al., 2020). We additionally implemented the following naive baseline (Naive OIE): each verb (detected using spaCy's POS-tagger (Honnibal and Montani, 2017)) becomes the predicate, its entire preceding sentence context becomes the subject and succeeding context the object. For ZH and DE , we evaluated a supervised M 2 OIE (Ro et al., 2020) model based on the multilingual BERT (Devlin et al., 2019), trained on a large EN dataset (Zhan and Zhao, 2020) and transferred (zero-shot) to target languages by means of its multilingual encoder. Implicit and N-ary Extractions. Some OIE systems produce implicit extractions containing tokens that do not occur in the sentence. 10 As BenchIE does not contain implicit annotations, we remove such extractions from the OIE systems' output, to avoid penalizing OIE systems for extracting fact types not covered by the benchmark. To make CaRB directly comparable, we automati-9 CaRB is an improved version of the widely-adopted OIE2016 benchmark (Stanovsky and Dagan, 2016); our findings for CaRB are thus likely to hold for OIE2016 as well. 10 E.g., the triple ("Biden"; "be"; "President") extracted from the phrase "President Biden ..." cally remove all its implicit extractions too. ROIE and M 2 OIE produce N-ary extractions (i.e., more than three slots), whereas BenchIE contains only triples. We follow standard practice (Del Corro and Gemulla, 2013) and convert those extractions into triples by concatenating the third and subsequent slots into a single object. Table 5 summarizes results of OIE systems on BenchIE and CaRB. Across the board, BenchIE's fact-level precision and recall are significantly lower than CaRB's respective precision and recall computed on token level. On average, CaRB scores the OIE systems higher than BenchIE by 14 percentage points for precision, 38 percentage points for recall and 26 percentage points for the F 1 score. Precision. System's precision on BenchIE is lower (albeit not so drastically lower as recall) than on CaRB because BenchIE, as a complete benchmark, punishes incorrect facts , i.e., extractions that cannot be found in BenchIE's fact synsets. CaRB, on the other hand, rewards any token overlap that the incorrectly extracted fact has against its gold triple(s)  in many cases such overlap is substantial and CaRB consequently rewards the incorrect fact with high precision. Consider, for example, the sentence from Table 1 and an incorrect fact extraction ( Sen. Mitchell ; is confident he has ; sufficient actions ); on BenchIE, this extraction is a false positive because it does not exist in any of the four fact synsets it lists for the sentence. CaRB, in contrast, rewards the extraction with perfect precision because all its tokens are accounted for in the corresponding slots of its gold triple ( Sen. Mitchell ; is confident he has" ; "sufficient votes to . . . actions ). In an attempt to quantify how much CaRB overestimates fact-level precision with its token overlap metric, we evaluated our Naive OIE baseline on both CaRB and BenchIE. While BenchIE reflects the poor quality of naive extractions with the near-zero performance, CaRB estimates its precision to be non-negligible (0.24) and even higher than that of the Stanford's OIE system (0.17). In contrast, BenchIE assigns much lower score to this baseline: precision of 0.038 times less than CaRB's score. Recall. While CaRB somewhat overestimates fact-level precision of OIE systems, its overestimation of their recall is much more drastic: all tokens of its gold extractions that can be found in respective slots of a factually incorrect extraction of an OIE system contribute to the system's recall. The overestimation of CaRB's recall scores is best illustrated by the fact that our naive baseline (Naive OIE) obtains a score of 0.7, better than any of the six OIE systems under evaluation. In terms of recall, CaRB obviously rewards long extractions  the longer the system extraction is, the more likely it is to cover more tokens from gold standard extractions. Neural extractors OpenIE6, ROIE, and M 2 OIE on average produce much longer extractions than rule-based systems like MinIE or Stanford (e.g., on average, a ROIE extraction has 16 tokens, whereas Stanford extraction has 7.7 tokens): accordingly, CaRB rewards the neural systems with much higher recall scores. BenchIE, on the other hand, credits only the OIE extractions that cover its fact synsets (and only once per fact synset). Our Naive OIE is, intuitively, highly unlikely to match gold extractions from fact synsets and BenchIE reflects this with a fact-level recall of only 2%. Similarly, BenchIE's recall scores reveal that the long extractions of neural OIE systems very rarely correspond to any acceptable variant of an expressed fact (e.g., ROIE's fact-level recall is only 9%). Multilingual OIE. We evaluated M 2 OIE (as the only multilingual model in our evaluation) on the Chinese and German versions of BenchIE. Quite expectedly, the performance for Chinese and German in target languages is below the source English performance. However, the drop due to the zero-shot language transfer is, at first glance  surprisingly, much larger for German than for Chinese: this goes against findings from other tasks, where transfer performance correlates with linguistic proximity between the source and target language (Lauscher et al., 2020). M 2 OIE's Chinese N a i v e OIEC l a u s IEM i n IES t a n f o r d ROIEO p e n IE 6 M 2 OIE ( EN ) 0.0 0.2 0.4 0.6 0.8 subject relation object Figure 1: Relative proportion of errors per slot for OIE systems. performance is encouraging, as it surpasses the English performance of some of the other OIE models (e.g., its recall score is better than ROIE, and its precision score is better than Stanford's). We believe this is because (a) OIE is a highly syntactic task; and (b) Chinese language is syntactically simple and has the same word order as English (SVO). German language, on the other hand, despite overall linguistic proximity to English, has a different word order (SOV; from generative per-spective), with the main verb often appearing at the very end of the sentence  this, we believe, is the main cause of poor OIE transfer between English and German. We believe BenchIE is a good starting point for multilingual OIE evaluation: we subsequently created additional data for Arabic, Galician, and Japanese: see Kotnis et al. (2022) and Friedrich et al. (2022) for details and further analyses. Token-based evaluation of existing OIE benchmarks (with real per-extraction scores in the range [0 , 1] ) makes pinpointing of extraction error source difficult. This limits their usability in automatic error analysis and system profiling. The fact that previous work performed OIE error analyses manually (Fader et al., 2011; Schneider et al., 2017) confirms this. BenchIE, in contrast, lists all acceptable extractions and thus naturally lends itself to reliable automatic error analysis and profiling. We carry out the analysis of errors per slots on the default BenchIE facet (2), because it is application-agnostic, unlike the additional facets from 3. We observed that most of the errors in all OIE systems stem from extracting the objects (see Figure 1). For an SVO language like English, 4478 ( 0 , 0 , 0 ) ( 0 , 0 , 1 ) ( 0 , 1 , 0 ) ( 0 , 1 , 1 ) ( 1 , 0 , 0 ) ( 1 , 0 , 1 ) ( 1 , 1 , 0 ) Naive OIE ClausIE MinIE Stanford ROIE OpenIE6 M2OIE (EN) 0.3 0.07 0.33 0.16 0.02 0 0.12 0.08 0.03 0.06 0.08 0.13 0.05 0.57 0.06 0.08 0.04 0.1 0.23 0.23 0.26 0.05 0.05 0.19 0.07 0.16 0.09 0.39 0.09 0.07 0.07 0.06 0.25 0.17 0.29 0.09 0.04 0.1 0.07 0.15 0.06 0.5 0.06 0.03 0.12 0.08 0.11 0.06 0.54 0.0 0.2 0.4 0.6 0.8 1.0 Figure 2: Distribution of incorrect extractions of OIE systems across different slot-error combinations. correctly extracting subjects and predicates seems substantially easier than correctly extracting objects. MinIE (rule-based) and ROIE (neural) have higher shares of predicate mis-extractions. MinIE post-processes ClausIE's triples by moving words from objects to predicates. Since ClausIE most frequently makes object errors, this effectively redistributes those errors between predicates and objects of MinIE's extractions. Figure 1, however, does not tell the whole story, as many extractions are erroneous in multiple slots. For more detailed insights, we assign each incorrect extraction to one of seven error buckets: each error bucket indicates one combination of extraction errors across the three slots. For example, the bucket (1 , 1 , 0) contains extractions that match their closest gold triple in the subject and predicate, but not object. The closest gold triple is the one that matches the extraction in most slots. 11 The error-bucket analysis, summarized in Figure 2, reveals that, across all systems, most extractions with object errors actually have correct subjects and predicates (bucket (1 , 1 , 0) ). MinIE deviates from this pattern and produces also many extractions with both incorrect object and predicate (bucket (1 , 0 , 0) ) or only bad predicate (bucket (1 , 0 , 1) ). Expectedly, most extractions of our naive baseline most often get only the predicate right (bucket (0 , 1 , 0) ) or all three slots wrong (bucket (0 , 0 , 0) ). This further emphasizes how misleading current token-based benchmarks can be  CaRB rewards this baseline with very high recall (see 4). To understand where OIE systems fail systematically, we split the input sentences into buckets and measured the performance of OIE systems per bucket. Based on preliminary qualitative error analysis, we chose bucketization according to some linguistic properties of the sentences that produced erroneous triples. In particular, we examine the performance of OIE systems for sentence length, presence of conjunctions and case markers, since these appeared to be the most common reasons for failure. Note that BenchIE instances can be bucke-tized according to an arbitrary dimension interest, lending itself to diverse future fine-grained evaluations and analyses of OIE systems' behaviour. In general, we found that OIE systems exhibit weakest performance on long sentences (with more than 30 tokens) as well as those that contain conjunctions or have more than two case markers (Figure 3). For a more detailed discussion, see Appendix C. 5.3 Multi-Faceted Evaluation Finally, we profile the OIE systems on our three special benchmark facets (3): BenchIE-E, -C and -M. Figure 4 summarizes the performance of OIE systems on these three facets. BenchIE-C. Ignoring slot boundaries, this facet is more lenient to OIE systems than the default facet  BenchIE-C yields higher scores than the regular BenchIE facet for all systems. The gap between the system's performance on BenchIE-C and BenchIE effectively quantifies how often the system misplaces tokens between adjacent slots. This gap is very small for Stanford OIE and MinIE  this means that, for extractions with correct overall token span, they also distribute the tokens between the slots correctly. For downstream tasks like text summarization, BenchIE-C results point to ClausIE as the best choice. Interestingly, we observed that CaRB's Precision for some systems (ClausIE and MinIE) effectively matches their Precision on BenchIE-C (see Figure 4), which is another indication that CaRB scores, in effect, neglect precise token distributions across slots. BenchIE-E. This facet is stricter than the default BenchIE facet  it allows fewer token placement variants in subject and object. For all OIE systems the F 1 BenchIE-E score is thus lower than the corresponding BenchIE score. MinIE and Stanford OIE obtain very similar performance on BenchIE-C, BenchIE (default), and BenchIE-E: this means that their extraction (when correct in overall token span) most often have clean concepts in subject and object. All neural systems and ClausIE exhibit huge performance drops on BenchIE-E  this 4479 Figure 3: Bucketized experiments: F1 score according to different bucketizations of the input sentences: sentence length means that their subject and object concept extractions are not clean, which makes these systems less suitable for tasks like KG population and entity linking. Out of the systems we evaluate, MinIE is the best fit for such downstream tasks. BenchIE-M. This facet yields the lowest performance for all systems, as it punishes extractions with any unnecessary tokens. Expectedly, MinIE  a system tailored to produce minimal extractions  yields the best performance on this facet. But even MinIE loses half of its performance when minimality is enforced (BenchIE vs. BenchIE-M). This calls for more work on minimizing OIE extractions. Stanford OIE outperforms all systems except MinIE, which renders it a good pick when extraction minimality is beneficial for a downstream task. Neural vs. Rule-Based Systems. Neural systems underperform their rule-based counterparts on most facets. This gap is most pronounced on BenchIE-E, whereas it is much smaller on BenchIE-C: these observations strongly indicate that neural systems struggle the most with correct distribution of tokens across the (adjacent) extraction slots. They also do not attempt to remove the optional (i.e., unnecessary) tokens, as indicated by extremely low performance on BenchIE-M. On CaRB, however, these same neural systems yield the best performance. Being trained and validated on datasets with extractions similar to CaRB's, neural extractors seem to overfit to CaRB evaluation. Our fact-based multi-faceted evaluation, however, reveals that their extractions are far less likely to be useful down the stream. We introduced BenchIE: a benchmark for more reliable fact-level evaluation of OIE systems for English, Chinese and German. Unlike existing benchmarks, BenchIE takes into account fact-level equivalence of extractions: it consists of fact synsets that contain all acceptable surface forms of the same fact. Further, EN BenchIE is multi-faceted  it allows to evaluate OIE extractions w.r.t. several aspects relevant in common downstream tasks. Our experiments show that current benchmarks, with incomplete gold standard and approximate token-level matching, drastically overestimate fact extraction abilities of OIE systems. Currently, the limits of BenchIE are its relatively small size (300 sentences v.s. CaRB's 1,200) and its time-consuming annotation process. A promising research direction is the investigation of trade-off between the manual effort and completeness of different OIE annotation strategies. In this scenario, BenchIE is an ideal point of reference: it can precisely quantify the completeness of some larger (non-exhaustive) OIE dataset created with limited or no manual effort.
abstain abstain abstain abstain abstain method abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain other method abstain other method abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain other abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain method other abstain abstain abstain other abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain