title: KITAB: Evaluating LLMs on Constraint Satisfaction for Information
    Retrieval
summary: We study the ability of state-of-the art models to answer constraint
  satisfaction queries for information retrieval (e.g., 'a list of ice cream
  shops in San Diego'). In the past, such queries were considered to be tasks
  that could only be solved via web-search or knowledge bases. More recently,
  large language models (LLMs) have demonstrated initial emergent abilities in
  this task. However, many current retrieval benchmarks are either saturated or
  do not measure constraint satisfaction. Motivated by rising concerns around
  factual incorrectness and hallucinations of LLMs, we present KITAB, a new
  dataset for measuring constraint satisfaction abilities of language models.
  KITAB consists of book-related data across more than 600 authors and 13,000
  queries, and also offers an associated dynamic data collection and constraint
  verification approach for acquiring similar test data for other authors. Our
  extended experiments on GPT4 and GPT3.5 characterize and decouple common
  failure modes across dimensions such as information popularity, constraint
  types, and context availability. Results show that in the absence of context,
  models exhibit severe limitations as measured by irrelevant information,
  factual errors, and incompleteness, many of which exacerbate as information
  popularity decreases. While context availability mitigates irrelevant
  information, it is not helpful for satisfying constraints, identifying
  fundamental barriers to constraint satisfaction. We open source our
  contributions to foster further research on improving constraint satisfaction
  abilities of future models.
paper_authors._id: 65386075a5ab055cf2d101fc
paper_authors.name: Marah I Abdin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-26T07:33:42.440Z
paper_authors._id: 649af1e3ca20306aeef590b8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649af1e3ca20306aeef590b8/Tvy8yf7zDO2FoNs03tWj6.jpeg
paper_authors.fullname: Marah Abdin
paper_authors.type: user
paper_authors.user: marah-abdin
paper_authors._id: 65386075a5ab055cf2d101fd
paper_authors.name: Suriya Gunasekar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T11:48:43.000Z
paper_authors._id: 63e1b4f77fbb6ae4d4f36aa4
paper_authors.avatarUrl: /avatars/7febc332d393e0bb94255104527ffe2a.svg
paper_authors.fullname: Gunasekar
paper_authors.type: user
paper_authors.user: suriyagunasekar
paper_authors._id: 65386075a5ab055cf2d101fe
paper_authors.name: Varun Chandrasekaran
paper_authors._id: 65386075a5ab055cf2d101ff
paper_authors.name: Jerry Li
paper_authors._id: 65386075a5ab055cf2d10200
paper_authors.name: Mert Yuksekgonul
paper_authors._id: 65386075a5ab055cf2d10201
paper_authors.name: Rahee Ghosh Peshawaria
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T11:49:42.221Z
paper_authors._id: 6529c0f3ff2202020e7851d0
paper_authors.avatarUrl: /avatars/05fa3c902862b22c7bf506a664736dd7.svg
paper_authors.fullname: Rahee Peshawaria
paper_authors.type: user
paper_authors.user: raghosh
paper_authors._id: 65386075a5ab055cf2d10202
paper_authors.name: Ranjita Naik
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T11:49:49.303Z
paper_authors._id: 63d8512207cd1aa3c4a1a835
paper_authors.avatarUrl: /avatars/cd5056c920a6b18eb93e2e9d0c5dc7c4.svg
paper_authors.fullname: Ranjita Naik
paper_authors.type: user
paper_authors.user: ranjita
paper_authors._id: 65386075a5ab055cf2d10203
paper_authors.name: Besmira Nushi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T11:49:56.490Z
paper_authors._id: 638ee2ed848625f9fad11034
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1670309301345-638ee2ed848625f9fad11034.jpeg
paper_authors.fullname: Besmira Nushi
paper_authors.type: user
paper_authors.user: nushib
paper_title: KITAB: Evaluating LLMs on Constraint Satisfaction for Information
    Retrieval
paper_summary: We study the ability of state-of-the art models to answer constraint
  satisfaction queries for information retrieval (e.g., 'a list of ice cream
  shops in San Diego'). In the past, such queries were considered to be tasks
  that could only be solved via web-search or knowledge bases. More recently,
  large language models (LLMs) have demonstrated initial emergent abilities in
  this task. However, many current retrieval benchmarks are either saturated or
  do not measure constraint satisfaction. Motivated by rising concerns around
  factual incorrectness and hallucinations of LLMs, we present KITAB, a new
  dataset for measuring constraint satisfaction abilities of language models.
  KITAB consists of book-related data across more than 600 authors and 13,000
  queries, and also offers an associated dynamic data collection and constraint
  verification approach for acquiring similar test data for other authors. Our
  extended experiments on GPT4 and GPT3.5 characterize and decouple common
  failure modes across dimensions such as information popularity, constraint
  types, and context availability. Results show that in the absence of context,
  models exhibit severe limitations as measured by irrelevant information,
  factual errors, and incompleteness, many of which exacerbate as information
  popularity decreases. While context availability mitigates irrelevant
  information, it is not helpful for satisfying constraints, identifying
  fundamental barriers to constraint satisfaction. We open source our
  contributions to foster further research on improving constraint satisfaction
  abilities of future models.
paper_ai_keywords: constraint satisfaction queries large language models (LLMs) factual incorrectness hallucinations KITAB feature representations sub-feature representations dynamic data collection constraint verification GPT4 GPT3.5 information popularity constraint types context availability irrelevant information factual errors incompleteness