title: Narrowing the Knowledge Evaluation Gap: Open-Domain Question Answering
    with Multi-Granularity Answers
summary: Factual questions typically can be answered correctly at different levels of
  granularity. For example, both ``August 4, 1961'' and ``1961'' are correct
  answers to the question ``When was Barack Obama born?''. Standard question
  answering (QA) evaluation protocols, however, do not explicitly take this into
  account and compare a predicted answer against answers of a single granularity
  level. In this work, we propose GRANOLA QA, a novel evaluation setting where a
  predicted answer is evaluated in terms of accuracy and informativeness against
  a set of multi-granularity answers. We present a simple methodology for
  enriching existing datasets with multi-granularity answers, and create
  GRANOLA-EQ, a multi-granularity version of the EntityQuestions dataset. We
  evaluate a range of decoding methods on GRANOLA-EQ, including a new algorithm,
  called Decoding with Response Aggregation (DRAG), that is geared towards
  aligning the response granularity with the model's uncertainty. Our experiments
  show that large language models with standard decoding tend to generate
  specific answers, which are often incorrect. In contrast, when evaluated on
  multi-granularity answers, DRAG yields a nearly 20 point increase in accuracy
  on average, which further increases for rare entities. Overall, this reveals
  that standard evaluation and decoding schemes may significantly underestimate
  the knowledge encapsulated in LMs.
paper_authors._id: 659e0b010ce6bc9fbd08909b
paper_authors.name: Gal Yona
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-10T09:36:27.520Z
paper_authors._id: 64637577c615cbc124477498
paper_authors.avatarUrl: /avatars/6bd01b4a6ea1e3a51d739d4104eb4ca9.svg
paper_authors.fullname: Gal Yona
paper_authors.type: user
paper_authors.user: galyona
paper_authors._id: 659e0b010ce6bc9fbd08909c
paper_authors.name: Roee Aharoni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-10T09:36:34.010Z
paper_authors._id: 6363bf2b123a5d5cd4a8fe7c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1667756718733-6363bf2b123a5d5cd4a8fe7c.jpeg
paper_authors.fullname: Roee Aharoni
paper_authors.type: user
paper_authors.user: roeeaharoni
paper_authors._id: 659e0b010ce6bc9fbd08909d
paper_authors.name: Mor Geva
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-10T09:36:41.188Z
paper_authors._id: 610b729f9da682cd54ad9adf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1628140189042-noauth.jpeg
paper_authors.fullname: Mor Geva
paper_authors.type: user
paper_authors.user: mega
paper_title: Narrowing the Knowledge Evaluation Gap: Open-Domain Question Answering
    with Multi-Granularity Answers
paper_summary: Factual questions typically can be answered correctly at different levels of
  granularity. For example, both ``August 4, 1961'' and ``1961'' are correct
  answers to the question ``When was Barack Obama born?''. Standard question
  answering (QA) evaluation protocols, however, do not explicitly take this into
  account and compare a predicted answer against answers of a single granularity
  level. In this work, we propose GRANOLA QA, a novel evaluation setting where a
  predicted answer is evaluated in terms of accuracy and informativeness against
  a set of multi-granularity answers. We present a simple methodology for
  enriching existing datasets with multi-granularity answers, and create
  GRANOLA-EQ, a multi-granularity version of the EntityQuestions dataset. We
  evaluate a range of decoding methods on GRANOLA-EQ, including a new algorithm,
  called Decoding with Response Aggregation (DRAG), that is geared towards
  aligning the response granularity with the model's uncertainty. Our experiments
  show that large language models with standard decoding tend to generate
  specific answers, which are often incorrect. In contrast, when evaluated on
  multi-granularity answers, DRAG yields a nearly 20 point increase in accuracy
  on average, which further increases for rare entities. Overall, this reveals
  that standard evaluation and decoding schemes may significantly underestimate
  the knowledge encapsulated in LMs.
paper_ai_summary: GRANOLA QA evaluates model answers against multi-granularity responses, revealing that a new decoding method improves accuracy and highlights limitations in standard evaluation methods.
paper_ai_keywords: GRANOLA QA multidisciplinary evaluation multi-granularity answers decoding methods Decoding with Response Aggregation (DRAG) large language models standard decoding accuracy rare entities