title: Investigating Answerability of LLMs for Long-Form Question Answering
summary: As we embark on a new era of LLMs, it becomes increasingly crucial to
  understand their capabilities, limitations, and differences. Toward making
  further progress in this direction, we strive to build a deeper understanding
  of the gaps between massive LLMs (e.g., ChatGPT) and smaller yet effective
  open-source LLMs and their distilled counterparts. To this end, we specifically
  focus on long-form question answering (LFQA) because it has several practical
  and impactful applications (e.g., troubleshooting, customer service, etc.) yet
  is still understudied and challenging for LLMs. We propose a
  question-generation method from abstractive summaries and show that generating
  follow-up questions from summaries of long documents can create a challenging
  setting for LLMs to reason and infer from long contexts. Our experimental
  results confirm that: (1) our proposed method of generating questions from
  abstractive summaries pose a challenging setup for LLMs and shows performance
  gaps between LLMs like ChatGPT and open-source LLMs (Alpaca, Llama) (2)
  open-source LLMs exhibit decreased reliance on context for generated questions
  from the original document, but their generation capabilities drop
  significantly on generated questions from summaries -- especially for longer
  contexts (>1024 tokens)
paper_authors._id: 65079e401704c7eb0a9b6273
paper_authors.name: Meghana Moorthy Bhat
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-18T19:43:01.520Z
paper_authors._id: 650889c4ce81b73e633d3a31
paper_authors.avatarUrl: /avatars/67572890f939149ccc6f078786e0ab8f.svg
paper_authors.fullname: Meghana Bhat
paper_authors.type: user
paper_authors.user: meghanabhat
paper_authors._id: 65079e401704c7eb0a9b6274
paper_authors.name: Rui Meng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-18T19:42:56.989Z
paper_authors._id: 61b022b39f13f47207d335cf
paper_authors.avatarUrl: /avatars/9cf00a775b80230aa32ef68ea7377833.svg
paper_authors.fullname: Rui
paper_authors.type: user
paper_authors.user: memray
paper_authors._id: 65079e401704c7eb0a9b6275
paper_authors.name: Ye Liu
paper_authors._id: 65079e401704c7eb0a9b6276
paper_authors.name: Yingbo Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-18T10:11:08.107Z
paper_authors._id: 649bc93758d8b19de0c7785f
paper_authors.avatarUrl: /avatars/3ed9473aee23d99f4ee949d3705089ea.svg
paper_authors.fullname: Yingbo Zhou
paper_authors.type: user
paper_authors.user: yingbozhou
paper_authors._id: 65079e401704c7eb0a9b6277
paper_authors.name: Semih Yavuz
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-18T10:11:16.798Z
paper_authors._id: 64f8a68e33fd3f0fcb7810b5
paper_authors.avatarUrl: /avatars/cb5ea85e9ed513efaf24a9f36869f98f.svg
paper_authors.fullname: Semih Yavuz
paper_authors.type: user
paper_authors.user: syavuz
paper_title: Investigating Answerability of LLMs for Long-Form Question Answering
paper_summary: As we embark on a new era of LLMs, it becomes increasingly crucial to
  understand their capabilities, limitations, and differences. Toward making
  further progress in this direction, we strive to build a deeper understanding
  of the gaps between massive LLMs (e.g., ChatGPT) and smaller yet effective
  open-source LLMs and their distilled counterparts. To this end, we specifically
  focus on long-form question answering (LFQA) because it has several practical
  and impactful applications (e.g., troubleshooting, customer service, etc.) yet
  is still understudied and challenging for LLMs. We propose a
  question-generation method from abstractive summaries and show that generating
  follow-up questions from summaries of long documents can create a challenging
  setting for LLMs to reason and infer from long contexts. Our experimental
  results confirm that: (1) our proposed method of generating questions from
  abstractive summaries pose a challenging setup for LLMs and shows performance
  gaps between LLMs like ChatGPT and open-source LLMs (Alpaca, Llama) (2)
  open-source LLMs exhibit decreased reliance on context for generated questions
  from the original document, but their generation capabilities drop
  significantly on generated questions from summaries -- especially for longer
  contexts (>1024 tokens)
paper_ai_summary: A question-generation method using abstractive summaries reveals performance gaps between large and small LLMs in long-form question answering tasks.
paper_ai_keywords: abstractive summaries long-form question answering (LFQA) long contexts