Retrieval is a core component for open-domain NLP tasks. In open-domain tasks, multiple entities can share a name, making disambiguation an inherent yet under-explored problem. We propose an evaluation benchmark for assessing the entity disambiguation capabilities of these retrievers, which we call Amb iguous E ntity R etrieval (AmbER) sets . We define an AmbER set as a collection of entities that share a name along with queries about those entities. By covering the set of entities for polysemous names, AmbER sets act as a challenging test of entity disambiguation. We create AmbER sets for three popular open-domain tasks: fact checking, slot filling, and question answering, and evaluate a diverse set of retrievers. We find that the retrievers exhibit popularity bias, sig-nificantly under-performing on rarer entities that share a name, e.g., they are twice as likely to retrieve erroneous documents on queries for the less popular entity under the same name. These experiments on AmbER sets show their utility as an evaluation tool and highlight the weaknesses of popular retrieval systems. 1 1 Introduction Substantial progress in NLP has been made on closed tasks, where queries are paired with relevant documents (Rajpurkar et al., 2016; Dua et al., 2019). However, there is growing interest in open-domain tasks, where relevant documents need to be retrieved from a knowledge source before an NLP system can perform reasoning and produce an answer (Chen et al., 2017; Petroni et al., 2021). The open-domain setting better reflects real-world usage for tasks where relevant information is generally not provided ( e.g., fact checking). Work started during an internship at Apple. 1 The AmbER sets used in this paper and the code to generate them are available at https://github.com/ anthonywchen/AmbER-Sets . Because success hinges on finding relevant documents, open-domain progress has been closely tied to improvements in retrieval systems 2 (Lee et al., 2019; Karpukhin et al., 2020; Lewis et al., 2020b). A crucial challenge when interacting with a large knowledge source ( e.g. , Wikipedia) is entity ambiguity, the phenomenon where a single name can map to multiple entities. Resolving this ambiguity is referred to as entity disambiguation and is an important step for effective retrieval. For example, given the query What musical instrument does Abe Lincoln play? , documents about the musician should rank higher than other entities with the same name (Figure 1). Although entity disambiguation has been extensively studied in entity linking (Hof-fart et al., 2011; Rao et al., 2013; Sevgili et al., 2 For example, replacing the BM25 retriever with DPR on Natural Questions increases exact match by 15 points. 2020) and search (Balog et al., 2010, 2011), in the context of open-domain NLP, it is unclear how good retrieval systems are when faced with queries with ambiguous entities. Evaluating entity ambiguity is challenging because the popularity of entities follows a long-tail (Figure 2) and rare entities are seldom covered in naturally-occurring datasets. In this paper we introduce AmbER sets, a benchmark for evaluating the entity disambiguation capabilities of retrievers across multiple NLP tasks. Each AmbER set is a collection of Wikidata entities that share a name, and their corresponding queries for specific NLP tasks. For each set, we define the head entity as the most popular entity and tail entities as the less popular ones. By creating queries for multiple entities that share a name, AmbER sets provide an accurate test of entity disambiguation capabilities of retrievers and help assess the role of entity popularity in disambiguation. We show examples of AmbER sets for the question answering task in Table 1. We automatically create AmbER sets by mining the Wikidata knowledge graph (Vrandecic and Krotzsch, 2014) for relevant names and entities, and leveraging task-specific templates to generate inputs for three tasks: fact checking, slot filling, and question answering (Figure 3). In total, our AmbER sets contain 80k task-specific queries which we align to the Wikipedia snapshot from KILT (Petroni et al., 2021). We use AmbER sets to conduct a systematic study of various retrieval systems that operate under different principles, such as token overlap and dense embedding similarity. Retrievers perform very differently on AmbER sets in terms of ab-solute retrieval numbers, with Bootleg (Orr et al., 2020), an entity-linking-based retriever, performing best. Despite these differences, all retrievers exhibit a large degree of popularity bias, under-performing on inputs concerning tail entities. TF-IDF, a token-based retriever, performs about four times worse on tail entity inputs compared to head entity inputs. Even with Bootleg, the best performing retriever, performance on tail entities is still 1.5 times lower than on head entities. Our results on AmbER sets demonstrate that there is significant work to be done on making retrievers robust in handling entity disambiguation. Retrieving relevant documents from large knowledge sources such as Wikipedia is an important first step in the open-domain pipeline. An inherent problem in working with such sources is entity disambiguation: resolving a name (mention) to an entity in the knowledge source. Entity disambiguation can be challenging because many entities share a name, and the popularity of entities follows a long-tail distribution (Figure 2). Despite the importance of entity disambiguation, it remains an understudied problem for open-domain NLP. We introduce AmbER sets for evaluating entity disambiguation capabilities of retrievers and analyze the role of entity popularity in disambiguation. We first provide an intuition for an AmbER set before concretely defining one. Consider two entities, a president and a musician, both of which have the name Abe Lincoln (Figure 1). Now, consider the query Which battle did Abe Lincoln fight in? and assume a retriever correctly returns the article about the president for this query. Simply because the correct document was retrieved does not mean a retriever has the ability to disambiguate between the president and the musician, as the president is much more popular. We should only be confident in its ability to disambiguate entities if we also pose a query about the less popular musician and the retriever again returns the correct document (as opposed to the document about the president). Based on this intuition, we define an AmbER set as a collection of queries that satisfy the following:  Criteria 1: Polysemous Name : The queries in an AmbER set are all about entities that share a common name ( e.g. , Abe Lincoln). Criteria 2: Disparity in Popularity : An AmbER set contains queries about both the most popular entity for a name (the head entity), e.g. , the president, and the less popular entities (the tail entities), e.g. , the musician. Criteria 3: Resolvable Ambiguity : The content of the query should be sufficient to resolve to the correct entity. The query Which battle did Abe Lincoln fight in? satisfies this criteria, because there is only one Abe Lincoln that fought in a war, while Where was Abe Lincoln born? does not since it applies to all Abe Lincolns. We provide examples of AmbER sets for the task of question answering in Table 1. 2.2 Open-Domain Tasks In this work, we create AmbER sets for three tasks: fact checking, slot filling, and question answering (Table 2). We consider these three tasks for three reasons. First, these three set of tasks are diverse in nature. In this work, slot filling is a generation task, question answering is a span selection task, and fact checking is a classification task. Second, the training sets available for each task are quite disparate. The largest fact checking training set, FEVER (Thorne et al., 2018), has 80k instances, while the slot filling dataset, T-REx (Elsahar et al., 2018), has over 2 million instances. The final reason we study these three tasks is that their inputs are short and easy to create. While AmbER sets can be manually created, doing so can be time-consuming, requiring a human to manually scour a knowledge base for polysemous names and related entities before manually writing queries for those entities. Instead, we present a pipeline for automatically creating AmbER sets using the Wikidata knowledge graph (Vrandecic and Krotzsch, 2014). In this section, we describe two different collections of AmbER sets, and discuss our automatic pipeline for creating AmbER sets. A natural question is How do retrievers handle entity ambiguity when two entities have the same entity type as opposed when they have different types?. To answer this question, we create two collections of AmbER sets. The first is AmbER-H , a collection of AmbER sets where all entities are humans. The choice to restrict AmbERH to humans is motivated by the fact that humans have properties that help distinguish themselves from other humans, generally based on occupation. The second is AmbERN , a collection of AmbER sets where all entities contained are non-humans, and disambiguation of a name is between non-human entities with different entity types. This is because a non-human entity, like a movie, does not generally have a single distinguishing property to distinguish from other movies. This makes it natural to compare non-human entities to other non-human entities with different types. We specify the entity types in each collection in Table 3. Davy Jones Name David Bowie* Popularity: 4.09 Wikidata Entities Davy Jones (racing driver) Popularity: 2.49 Davy Jones (baseball) Popularity: 1.93 Gender: Male Birthplace : Brixton Gender : Male Sport : Baseball Gender : Male Sport : Auto Racing Movement : New Wave Wikidata Properties Sports Team : Chicago White Sox Task Specific Inputs QA: Which movement is Davy Jones associated with? We now describe a pipeline to automatically create AmbER sets for three tasks: fact checking, slot filling, and question answering. We provide a visualization of the pipeline in Figure 3. Collecting Names and Entities We begin by collecting all entity aliases 3 in Wikidata. From these aliases, we filter for those that are shared by multiple Wikidata entities. Each entity in Wikidata is represented by a unique QID. The entities must have an entity type from Table 3 depending on the collection we are collecting AmbER sets for. Each alias and associated entities form the basis for an AmbER set. Within each set, we define the head and tail entities based on the number of Wikipedia page views for the month of October 2019. We filter out AmbER sets where the percentage gap in popularity between the head entity and the most popular tail entity is less than 10% to account for noise in the monthly page views. Collecting Distinguishing Properties We gather properties and associated values for each entity from Wikidata. We only retain properties that are in a specified list (Table 3), as they are useful for resolving ambiguity (Criteria 3) . We also filter a property if two entities within an AmbER set have that property, ensuring that the remaining properties can be used to disambiguate between entities with the same name. These properties are used to instantiate the queries. 3 Aliases are all possible names for an entity. the knowledge source for AmbER sets for better reproducibility. Each Wikipedia document in KILT has an associated QID. For each entity, we find all Wikipedia documents with that associated QID. After this alignment, we apply a round of filtering on the tuples. For each tuple, we check that the value of the tuple is within the first 350 tokens of the aligned Wikipedia article. If not, we remove AmbERH AmbERN # AmbER Sets 2,093 5,237 Averages per AmbER Set ...# entities 2.98 2.42 ...# entities w/ properties 2.03 2.06 ...# properties 2.84 2.64 # Input Queries 23,768 55,216 ...Question Answering (QA) 5,942 13,804 ...Slot Filling (SF) 5,942 13,804 ...Fact checking (FC) 11,884 27,608 Table 4: Statistics of AmbER collections. Instantiating AmbER Instances Recall that our goal was to create AmbER sets for three tasks: fact checking, slot filling, and question answering. We are able to create queries for all three tasks simultaneously using the collected Wikidata tuples. For question answering and fact checking, we use templates based on properties to instantiate inputs. Three of the authors wrote a template each for each property for the two tasks. Duplicate templates are removed, resulting in an average of 3 question answering templates per property and 2.7 fact checking templates per property. See Appendix B for the complete list of templates. For slot filling, we create a single input from each Wikidata tuple by concatenating the AmbER set name with the property name, and using the value of the tuple as the answer. For question answering, we also create a single input for each tuple by filling in the template with the AmbER set name and using the value of the tuple as the answer. For fact checking, we create two inputs for each tuple, one claim that is true using the tuple value and one claim that is false. The false claim is created by finding the most popular value for the tuple property that does not match the tuple value 5 . We provide statistics for AmbER sets in Table 4. On average, each AmbER set has about three entities that share the same name. Of these three entities, on average, only two have properties after filtering. In total, our AmbER sets contain about 80k task-specific input queries. 4 This reduces the number of tuples for AmbERH from 17,079 to 5,942 and for AmbERN from 22,219 to 13,804. Since our pipeline is automated and relies on Wikipedia and Wikidata, there are a few limitations worth noting. AmbER sets will be affected by incompleteness of the knowledge source, sometimes resulting ambiguous queries if a property is missing from Wikidata, but answerable from Wikipedia text. For this reason, we only select a few properties for each type (Table 3). Second, even though we author multiple templates for each property, the reliance on these templates limits the syntactic diversity in the queries (not a critical concern, since we are only evaluating existing models). Also, we use Wikipedia page views as a proxy for real-world popularity of entities. Defining popularity in this way may be problematic, as page views for an entity can fluctuate, and may make our pipeline difficult to generalize to other knowledge sources, where this information may not be available. Several design choices in creating AmbER sets are worth further investigation. We limit AmbER sets to a pre-specified list of entity types and properties to ensure that entities in an AmbER set are distinguishable. This precludes other properties that may be useful in distinguishing entities, reducing the diversity in AmbER sets. Another design choice is we allow any alias in Wikidata to form an AmbER sets, however, not all aliases are canonical ways to refer to the entity. For instance, Shaquille O'Neal has the unusual alias The Big Cactus, potentially leading to a somewhat unrealistic query What sport did The Big Cactus play? . We plan to revisit the these design choices in future work. Retrieval Systems The primary focus of this work is to evaluate entity ambiguity of retrieval systems. We consider four retrievers based on different retrieval paradigms. The first three are TF-IDF, a token-based retriever using sparse em-beddings, DPR (Karpukhin et al., 2020), a dense embedding based retriever, and BLINK (Wu et al., 2020), a linker-based retriever which ranks documents based on input entities. These three retrievers have been thoroughly evaluated on a number of open-domain tasks in Petroni et al. (2021) with no obvious winner across tasks. Encouraged by the disambiguation success on rare entities by Orr et al. (2020), we also evaluate a retriever based on Bootleg, another entity linker. We provide additional details about these retrievers in Appendix D. Collection Retriever Fact Checking (FC) Slot Filling (SF) Question Answering (QA) All Head Tail  All Head Tail  All Head Tail  AmbERHTF-IDF 17.3 28.5 8.2 0.0 18.8 31.9 8.1 0.0 16.7 28.2 7.3 0.1 DPR 18.1 23.9 13.3 0.1 8.0 11.6 5.1 0.3 13.1 19.6 7.9 1.1 BLINK 55.9 64.4 49.0 5.6 38.2 57.0 22.9 11.5 31.7 40.5 24.6 6.6 Bootleg 34.8 43.0 28.2 0.7 56.5 63.9 50.6 25.3 67.2 77.1 59.1 36.1 AmbERNTF-IDF 9.4 13.6 4.9 0.0 13.4 21.0 5.2 0.2 13.9 21.7 5.4 0.3 DPR 36.9 48.0 24.8 4.4 29.9 40.9 18.0 6.0 36.2 49.2 22.2 9.3 BLINK 11.7 13.9 9.4 0.0 5.7 7.3 3.9 0.7 35.2 44.7 24.9 10.1 Bootleg 3.5 4.6 2.4 0.0 52.3 61.3 42.5 22.4 59.8 69.5 49.3 29.0 Table 5: Top-1 retrieval results on each collection of AmbER sets. Downstream Models The dominant approach to open-domain tasks is a two-stage process where a retriever first finds relevant documents, followed by a downstream model that processes these documents to produce an answer. We evaluate the end-to-end performance on AmbER sets by training downstream NLP models on our tasks of interest. For fact checking, we fine-tune a BERT classifier (Devlin et al., 2019) on FEVER (Thorne et al., 2018). For question answering, we fine-tune a RoBERTa model (Liu et al., 2019) on Natural Questions (Kwiatkowski et al., 2019). For slot filling, a generation task, we fine-tune a BART model (Lewis et al., 2020a) on T-Rex (Elsahar et al., 2018). We provide example training instances in Table 2 and additional details on the models in Appendix E. We use the AllenNLP and HuggingFace Transformers library to finetune our downstream models (Gardner et al., 2018; Wolf et al., 2020). In this section, we evaluate existing open-domain NLP pipelines using AmbER sets. We also conduct Figure 4: Popularity Gap vs Retrieval Gap. Top Document Retrieval We report retrieval performance in Table 5 in terms of retriever ac-curacy@1 (the % of instances where the first retrieved document is the gold document). For each task, we report values on the entire AmbER set (All), as well as instances corresponding only to Head entities or to Tail entities. We also report a metric we call all correct (  ), the fraction Task System Results All Head Tail HFC BERT (Oracle) 77.7 73.6 80.3 BERT + BLINK 59.8 60.1 57.7 SF BART (Oracle) 83.9 85.0 83.5 BART + BLINK 34.4 38.2 32.6 QA BERT (Oracle) 71.4 77.7 83.0 BERT + BLINK 27.5 33.8 22.3 NFC BERT (Oracle) 66.6 63.9 69.5 BERT + DPR 60.9 61.4 60.4 SF BART (Oracle) 82.1 80.1 84.3 BART + DPR 18.6 18.6 18.6 QA BERT (Oracle) 83.5 85.1 81.8 BERT + DPR 26.0 31.3 20.4 Table 7: End-to-end performance on AmbER sets . of AmbER sets in which all queries had the correct document retrieved. All retrievers do better on head entities compared to tail entities. Since BLINK, Bootleg, and DPR are initialized using pre-trained language models, they may have a predisposition towards being biased to more popular entities. However, we find TF-IDF also does better on head entities, perhaps because more popular entities have longer Wikipedia pages, possibly increasing term-frequency scores. Second, there are large discrepancies between a retriever's performance on different tasks for an AmbER collection. For instance, DPR does substantially worse on slot filling compared to its performance on question answering. This is surprising since queries for all tasks are created from the same set of Wikidata tuples. Finally, we find that retrievers are mostly incorrect on getting all the queries in a set correct, with some receiving a  score of 0 on some tasks. Overall, we find that the Bootleg retriever on average does the best across tasks, however there is significant scope for improvement. Entity Confusion To explicitly evaluate whether retrievers get confused by entities in the same AmbER set, we compute entity confusion for retrievers defined as the percentage of queries where the retriever ranks a document for an incorrect entity from the same AmbER set over the gold document (Table 6). We find that across retrievers, tasks, and AmbER collections, entity confusion is twice as high for tail entity inputs. This result indicates that the popularity of an entity for a given name plays a significant role in retrieval performance. Effect of Popularity Gap Since the difference in popularity between the head and tail entities can vary considerably, these results obfuscate the effect of the size of the popularity gap. We explore how the gap in popularity between head and tail entities translates to the gaps in performance on their associated queries. For a head entity with popularity p h and a tail entity with popularity p t from the same AmbER set, we calculate popularity gap, p h  p t p t , and bin associated head/tail inputs based on the gap 6 . For each bin, we calculate the difference in accuracy@1 between the head and tail entity queries. Results for QA AmbER sets (Figure 4) show that there is a strong correlation between the popularity gap and the difference in performance. End to End Results We evaluate end to end performance in several evaluation settings with all results provided in Table 7. The metrics used are F1 for slot filling and question answering and accuracy for fact checking. In the oracle setting, we directly provide the downstream NLP model the gold document, and find that the gap between head entities and tail entities is fairly small. This suggests that in closed NLP settings, where the gold document is known, entity disambiguation is not a major concern. In the regular retrieval setting, we provide the model the top 20 documents as ranked by a retrieval system (BLINK and DPR), and find that retrievers still perform better on head entity queries (see Appendix A). The downstream systems that use retrieved documents display a noticeable gap in end-to-end performance between head and tail entity inputs. This is expected, as retrieval systems perform worse on tail entities. User Study AmbER sets are created in a largely automatic process, raising questions about data quality. To address these questions, we conduct a small user study on AmbER sets to evaluate whether the queries are resolvable by humans. We present a query from a QA AmbER set along with three documents for the entities from the same AmbER set, one of which is the gold document. We first ask the user to select the relevant document, then we ask the user to select an answer span from the selected document. In total, we asked 7 subjects to examine about 120 queries across AmbER-H and AmbERN , and computed their accuracy in 6 Bin width of 20%. Queries with a popularity gap higher than 100% are binned into the highest bin. selecting the correct document and answer (Table 8). We also compare retrievers for this task, i.e. select from 3 documents for the same queries, and find that humans perform very well on the document selection task compared to retrievers on both sets. We also compare the accuracy of answer selection, and see that the closed domain NLP model (fine-tuned BERT) is as almost accurate as humans on the same set of queries 7 . This further confirms that closed NLP models are not the source of bias towards head entities, but the retrievers are. Entity Ambiguity As previously mentioned, entity ambiguity is when a single name can match multiple entities in a knowledge source. Entity ambiguity has been most studied in the context of entity linking (Rao et al., 2013). To improve disambiguation, entity linkers have included auxiliary information such as entity types (Onoe and Durrett, 2020) and entity descriptions (Logeswaran et al., 2019). A recent thread of work aims to study how language models recall and leverage information about names and entities. Prabhakaran et al. (2019) shows that names can have a measurable effect on the prediction of sentiment analysis systems. Shwartz et al. (2020) demonstrates that pre-trained language models implicitly resolve entity ambiguity by grounding names to entities based on the pretraining corpus. The problem of entity ambiguity also appears implicitly in entity-centric tasks such as determining the semantic relatedness between entities (Hoffart et al., 2012) and entity-oriented 7 The relatively low answer score is due to artifacts in using EM for QA evaluation, and is consistent with human performance on span selection (Rajpurkar et al., 2016)). Popularity Bias System's that perform worse on the long-tail suffer from what is known as popularity bias. This problem has been studied extensively in the recommendation systems literature, where recommendation systems are known to often ignore the long-tail of products and instead recommend very popular items (Abdollahpouri et al., 2017; Chen et al., 2020). This has the effect of unfairly hurting users who would prefer these less-popular items (Abdollahpouri et al., 2019; Ciampaglia et al., 2018). We explore popularity bias from the angle of retrieval as opposed to recommendation, and find popularity bias exists in retrieval systems. Open-Domain Ambiguity Ambiguity is an inherent problem when it comes to open-domain reasoning. Min et al. (2020) showed that half of instances sampled from Natural Questions are ambiguous, with multiple correct answers. AmbER sets are similar in that the ambiguity is in terms of the entity in the query, however, in contrast to Natural Questions, AmbER set inputs have been constructed such that the ambiguity is resolvable. Challenge Sets There have been many evaluation sets specifically designed to assess a model's ability to handle a specific phenomenon (Naik et al., 2018; Zhao et al., 2018; McCoy et al., 2019; Warstadt et al., 2020; Richardson et al., 2020; Jeretic et al., 2020; Ribeiro et al., 2019). Some of these challenge sets, similar to AmbER sets, use templates to generate a large amount of evaluation data quickly (Richardson et al., 2020; McCoy et al., 2019; Ribeiro et al., 2020). AmbER sets can be viewed as a challenge set for assessing open-domain systems' ability to handle entity ambiguity. Entity ambiguity is an inherent problem in retrieval, as many entities can share a name. For evaluating disambiguation capabilities of retrievers, we introduce AmbER sets; an AmbER set is a collection of task-specific queries about entities that share a name, but the queries have sufficient content to resolve the correct entity. We create a broad range of AmbER sets, covering many entity types, with input queries for three open-domain NLP tasks: fact checking, slot filling, and question answering. Our experiments demonstrate the struggles of current retrievers in handling entity ambiguity. In particular, we find that the popularity of an entity in relation to other entities that share a name plays a significant role during disambiguation. For instance, we find that all tested retrievers are about twice as likely to retrieve erroneous documents when dealing with less popular entities than the most popular entity with the same name. Future goals include improving entity disambiguation capabilities of retrievers, perhaps more directly incorporating ideas from entity linking and coreference resolution. The AmbER sets and the code for the generation pipeline is available at https: //github.com/anthonywchen/AmbER-Sets . We would like to thank Jo Daiber, Michael Tu, Russ Webb, Matt Gardner, Robert Logan, Sherry Tongshuang Wu, and the anonymous reviewers for providing valuable feedback for our work. This work is funded in part by the DARPA MCS program under Contract No. N660011924033 with the United States Office Of Naval Research.
abstain abstain objective method abstain method result abstain abstain abstain abstain abstain other abstain abstain abstain abstain other abstain method abstain method abstain result method abstain method abstain abstain abstain abstain objective abstain abstain abstain abstain abstain method method abstain abstain abstain method method abstain abstain abstain method method method abstain method abstain abstain method abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain method method method method abstain method abstain method method method abstain method abstain abstain abstain abstain abstain method abstain method abstain method method abstain abstain abstain method method method abstain method abstain abstain abstain abstain abstain abstain abstain method method method abstain method abstain method abstain abstain method method abstain abstain method method abstain method method method method method abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain result result abstain result abstain abstain abstain abstain abstain abstain result abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain other other other other other other other other other other objective other other other other other other abstain method method objective result result abstain other other other other