QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set
  Operations
Formulating selective information needs results in queries that implicitly
specify set operations, such as intersection, union, and difference. For
instance, one might search for "shorebirds that are not sandpipers" or
"science-fiction films shot in England". To study the ability of retrieval
systems to meet such information needs, we construct QUEST, a dataset of 3357
natural language queries with implicit set operations, that map to a set of
entities corresponding to Wikipedia documents. The dataset challenges models to
match multiple constraints mentioned in queries with corresponding evidence in
documents and correctly perform various set operations. The dataset is
constructed semi-automatically using Wikipedia category names. Queries are
automatically composed from individual categories, then paraphrased and further
validated for naturalness and fluency by crowdworkers. Crowdworkers also assess
the relevance of entities based on their documents and highlight attribution of
query constraints to spans of document text. We analyze several modern
retrieval systems, finding that they often struggle on such queries. Queries
involving negation and conjunction are particularly challenging and systems are
further challenged with combinations of these operations.
646aff1b35c7a57f936d46b1
Chaitanya Malaviya
claimed_verified
2023-05-22T18:01:21.634Z
646bad133e2a7b065948d4be
https://cdn-avatars.huggingface.co/v1/production/uploads/646bad133e2a7b065948d4be/Hk21DxumaBbkMJNZCKL4A.jpeg
Chaitanya Malaviya
user
cmalaviya
646aff1b35c7a57f936d46b2
Peter Shaw
claimed_verified
2023-05-22T18:01:25.347Z
646b85a4e96a751c5252ae41
/avatars/a47d5f2b17887e61716d0c180c95ae1f.svg
Peter Shaw
user
ptshaw
646aff1b35c7a57f936d46b3
Ming-Wei Chang
646aff1b35c7a57f936d46b4
Kenton Lee
646aff1b35c7a57f936d46b5
Kristina Toutanova
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set
  Operations
Formulating selective information needs results in queries that implicitly
specify set operations, such as intersection, union, and difference. For
instance, one might search for "shorebirds that are not sandpipers" or
"science-fiction films shot in England". To study the ability of retrieval
systems to meet such information needs, we construct QUEST, a dataset of 3357
natural language queries with implicit set operations, that map to a set of
entities corresponding to Wikipedia documents. The dataset challenges models to
match multiple constraints mentioned in queries with corresponding evidence in
documents and correctly perform various set operations. The dataset is
constructed semi-automatically using Wikipedia category names. Queries are
automatically composed from individual categories, then paraphrased and further
validated for naturalness and fluency by crowdworkers. Crowdworkers also assess
the relevance of entities based on their documents and highlight attribution of
query constraints to spans of document text. We analyze several modern
retrieval systems, finding that they often struggle on such queries. Queries
involving negation and conjunction are particularly challenging and systems are
further challenged with combinations of these operations.
A dataset named QUEST challenges retrieval systems by testing their ability to interpret natural language queries with implicit set operations and cross-reference them with Wikipedia documents.