title: QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set
    Operations
summary: Formulating selective information needs results in queries that implicitly
  specify set operations, such as intersection, union, and difference. For
  instance, one might search for "shorebirds that are not sandpipers" or
  "science-fiction films shot in England". To study the ability of retrieval
  systems to meet such information needs, we construct QUEST, a dataset of 3357
  natural language queries with implicit set operations, that map to a set of
  entities corresponding to Wikipedia documents. The dataset challenges models to
  match multiple constraints mentioned in queries with corresponding evidence in
  documents and correctly perform various set operations. The dataset is
  constructed semi-automatically using Wikipedia category names. Queries are
  automatically composed from individual categories, then paraphrased and further
  validated for naturalness and fluency by crowdworkers. Crowdworkers also assess
  the relevance of entities based on their documents and highlight attribution of
  query constraints to spans of document text. We analyze several modern
  retrieval systems, finding that they often struggle on such queries. Queries
  involving negation and conjunction are particularly challenging and systems are
  further challenged with combinations of these operations.
paper_authors._id: 646aff1b35c7a57f936d46b1
paper_authors.name: Chaitanya Malaviya
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-22T18:01:21.634Z
paper_authors._id: 646bad133e2a7b065948d4be
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646bad133e2a7b065948d4be/Hk21DxumaBbkMJNZCKL4A.jpeg
paper_authors.fullname: Chaitanya Malaviya
paper_authors.type: user
paper_authors.user: cmalaviya
paper_authors._id: 646aff1b35c7a57f936d46b2
paper_authors.name: Peter Shaw
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-22T18:01:25.347Z
paper_authors._id: 646b85a4e96a751c5252ae41
paper_authors.avatarUrl: /avatars/a47d5f2b17887e61716d0c180c95ae1f.svg
paper_authors.fullname: Peter Shaw
paper_authors.type: user
paper_authors.user: ptshaw
paper_authors._id: 646aff1b35c7a57f936d46b3
paper_authors.name: Ming-Wei Chang
paper_authors._id: 646aff1b35c7a57f936d46b4
paper_authors.name: Kenton Lee
paper_authors._id: 646aff1b35c7a57f936d46b5
paper_authors.name: Kristina Toutanova
paper_title: QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set
    Operations
paper_summary: Formulating selective information needs results in queries that implicitly
  specify set operations, such as intersection, union, and difference. For
  instance, one might search for "shorebirds that are not sandpipers" or
  "science-fiction films shot in England". To study the ability of retrieval
  systems to meet such information needs, we construct QUEST, a dataset of 3357
  natural language queries with implicit set operations, that map to a set of
  entities corresponding to Wikipedia documents. The dataset challenges models to
  match multiple constraints mentioned in queries with corresponding evidence in
  documents and correctly perform various set operations. The dataset is
  constructed semi-automatically using Wikipedia category names. Queries are
  automatically composed from individual categories, then paraphrased and further
  validated for naturalness and fluency by crowdworkers. Crowdworkers also assess
  the relevance of entities based on their documents and highlight attribution of
  query constraints to spans of document text. We analyze several modern
  retrieval systems, finding that they often struggle on such queries. Queries
  involving negation and conjunction are particularly challenging and systems are
  further challenged with combinations of these operations.
paper_ai_summary: A dataset named QUEST challenges retrieval systems by testing their ability to interpret natural language queries with implicit set operations and cross-reference them with Wikipedia documents.