title: Evaluating D-MERIT of Partial-annotation on Information Retrieval
summary: Retrieval models are often evaluated on partially-annotated datasets. Each
  query is mapped to a few relevant texts and the remaining corpus is assumed to
  be irrelevant. As a result, models that successfully retrieve false negatives
  are punished in evaluation. Unfortunately, completely annotating all texts for
  every query is not resource efficient. In this work, we show that using
  partially-annotated datasets in evaluation can paint a distorted picture. We
  curate D-MERIT, a passage retrieval evaluation set from Wikipedia, aspiring to
  contain all relevant passages for each query. Queries describe a group (e.g.,
  ``journals about linguistics'') and relevant passages are evidence that
  entities belong to the group (e.g., a passage indicating that Language is a
  journal about linguistics). We show that evaluating on a dataset containing
  annotations for only a subset of the relevant passages might result in
  misleading ranking of the retrieval systems and that as more relevant texts are
  included in the evaluation set, the rankings converge. We propose our dataset
  as a resource for evaluation and our study as a recommendation for balance
  between resource-efficiency and reliable evaluation when annotating evaluation
  sets for text retrieval.
paper_authors._id: 667a6c233f20f47d49334f91
paper_authors.name: Royi Rassin
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-06-25T07:05:08.346Z
paper_authors._id: 62a7581cf049be35252a2e7c
paper_authors.avatarUrl: /avatars/91de4eb48f51bfd6e028c08ccfa98f8c.svg
paper_authors.fullname: Royi Rassin
paper_authors.type: user
paper_authors.user: Royir
paper_authors._id: 667a6c233f20f47d49334f92
paper_authors.name: Yaron Fairstein
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T11:17:13.015Z
paper_authors._id: 63a417660cf4daf616676972
paper_authors.avatarUrl: /avatars/75cd5327713e078e5e70c8a8086f3488.svg
paper_authors.fullname: Yaron Fairstein
paper_authors.type: user
paper_authors.user: yyfairstein
paper_authors._id: 667a6c233f20f47d49334f93
paper_authors.name: Oren Kalinsky
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-06-25T07:05:08.346Z
paper_authors._id: 6306aed3cfbde33ef7dd6dac
paper_authors.avatarUrl: /avatars/a59f8ffec3b87d71e428425d74c74795.svg
paper_authors.fullname: Oren Kalinsky
paper_authors.type: user
paper_authors.user: orenk
paper_authors._id: 667a6c233f20f47d49334f94
paper_authors.name: Guy Kushilevitz
paper_authors._id: 667a6c233f20f47d49334f95
paper_authors.name: Nachshon Cohen
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-27T08:03:33.794Z
paper_authors._id: 63a417f9412fd71fb7ebd218
paper_authors.avatarUrl: /avatars/0dcac15e5cc6c9cb89dd7af2818bacb9.svg
paper_authors.fullname: Nachshon Cohen
paper_authors.type: user
paper_authors.user: nachshonc
paper_authors._id: 667a6c233f20f47d49334f96
paper_authors.name: Alexander Libov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T13:34:22.404Z
paper_authors._id: 63ad4bae9835dd4b124f32c5
paper_authors.avatarUrl: /avatars/972929b6e560f328a305f99e5365b685.svg
paper_authors.fullname: Alexander Libov
paper_authors.type: user
paper_authors.user: alibov
paper_authors._id: 667a6c233f20f47d49334f97
paper_authors.name: Yoav Goldberg
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-12-26T13:11:49.220Z
paper_authors._id: 676d5323dd95830fd94d675f
paper_authors.avatarUrl: /avatars/a8622e2a189a13c7b5a99177c3a87c37.svg
paper_authors.fullname: Yoav Goldberg
paper_authors.type: user
paper_authors.user: yoavgoldberg
paper_title: Evaluating D-MERIT of Partial-annotation on Information Retrieval
paper_summary: Retrieval models are often evaluated on partially-annotated datasets. Each
  query is mapped to a few relevant texts and the remaining corpus is assumed to
  be irrelevant. As a result, models that successfully retrieve false negatives
  are punished in evaluation. Unfortunately, completely annotating all texts for
  every query is not resource efficient. In this work, we show that using
  partially-annotated datasets in evaluation can paint a distorted picture. We
  curate D-MERIT, a passage retrieval evaluation set from Wikipedia, aspiring to
  contain all relevant passages for each query. Queries describe a group (e.g.,
  ``journals about linguistics'') and relevant passages are evidence that
  entities belong to the group (e.g., a passage indicating that Language is a
  journal about linguistics). We show that evaluating on a dataset containing
  annotations for only a subset of the relevant passages might result in
  misleading ranking of the retrieval systems and that as more relevant texts are
  included in the evaluation set, the rankings converge. We propose our dataset
  as a resource for evaluation and our study as a recommendation for balance
  between resource-efficiency and reliable evaluation when annotating evaluation
  sets for text retrieval.
paper_ai_keywords: passage retrieval evaluation partially-annotated datasets false negatives annotation relevant passages Wikipedia text retrieval computational rankings evaluation sets text retrieval