title: Unstructured Evidence Attribution for Long Context Query Focused
    Summarization
summary: Large language models (LLMs) are capable of generating coherent summaries
  from very long contexts given a user query. Extracting and properly citing
  evidence spans could help improve the transparency and reliability of these
  summaries. At the same time, LLMs suffer from positional biases in terms of
  which information they understand and attend to, which could affect evidence
  citation. Whereas previous work has focused on evidence citation with
  predefined levels of granularity (e.g. sentence, paragraph, document, etc.), we
  propose the task of long-context query focused summarization with unstructured
  evidence citation. We show how existing systems struggle to generate and
  properly cite unstructured evidence from their context, and that evidence tends
  to be "lost-in-the-middle". To help mitigate this, we create the Summaries with
  Unstructured Evidence Text dataset (SUnsET), a synthetic dataset generated
  using a novel domain-agnostic pipeline which can be used as supervision to
  adapt LLMs to this task. We demonstrate across 5 LLMs of different sizes and 4
  datasets with varying document types and lengths that LLMs adapted with SUnsET
  data generate more relevant and factually consistent evidence than their base
  models, extract evidence from more diverse locations in their context, and can
  generate more relevant and consistent summaries.
paper_authors._id: 67b83a20a9fa331061e84ecd
paper_authors.name: Dustin Wright
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T09:58:02.288Z
paper_authors._id: 60a643b9213fe60589b8fdf9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60a643b9213fe60589b8fdf9/OOXmW3MkSf88r63tAE6-n.jpeg
paper_authors.fullname: Dustin Wright
paper_authors.type: user
paper_authors.user: dwright37
paper_authors._id: 67b83a20a9fa331061e84ece
paper_authors.name: Zain Muhammad Mujahid
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:16:52.600Z
paper_authors._id: 637e8b1b66ee00bcb2468ed0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669240174964-637e8b1b66ee00bcb2468ed0.jpeg
paper_authors.fullname: Zain
paper_authors.type: user
paper_authors.user: zainmujahid
paper_authors._id: 67b83a20a9fa331061e84ecf
paper_authors.name: Lu Wang
paper_authors._id: 67b83a20a9fa331061e84ed0
paper_authors.name: Isabelle Augenstein
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:17:02.420Z
paper_authors._id: 608918b7df398c3b285ce960
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1621507769190-608918b7df398c3b285ce960.jpeg
paper_authors.fullname: Isabelle Augenstein
paper_authors.type: user
paper_authors.user: IAugenstein
paper_authors._id: 67b83a20a9fa331061e84ed1
paper_authors.name: David Jurgens
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:17:08.686Z
paper_authors._id: 63516acdce7cf1fe8a854cdc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63516acdce7cf1fe8a854cdc/TlOI7iPdG7zJKWyGLoPQN.jpeg
paper_authors.fullname: David Jurgens
paper_authors.type: user
paper_authors.user: davidjurgens
paper_title: Unstructured Evidence Attribution for Long Context Query Focused
    Summarization
paper_summary: Large language models (LLMs) are capable of generating coherent summaries
  from very long contexts given a user query. Extracting and properly citing
  evidence spans could help improve the transparency and reliability of these
  summaries. At the same time, LLMs suffer from positional biases in terms of
  which information they understand and attend to, which could affect evidence
  citation. Whereas previous work has focused on evidence citation with
  predefined levels of granularity (e.g. sentence, paragraph, document, etc.), we
  propose the task of long-context query focused summarization with unstructured
  evidence citation. We show how existing systems struggle to generate and
  properly cite unstructured evidence from their context, and that evidence tends
  to be "lost-in-the-middle". To help mitigate this, we create the Summaries with
  Unstructured Evidence Text dataset (SUnsET), a synthetic dataset generated
  using a novel domain-agnostic pipeline which can be used as supervision to
  adapt LLMs to this task. We demonstrate across 5 LLMs of different sizes and 4
  datasets with varying document types and lengths that LLMs adapted with SUnsET
  data generate more relevant and factually consistent evidence than their base
  models, extract evidence from more diverse locations in their context, and can
  generate more relevant and consistent summaries.
paper_ai_summary: LMMs adapted with a new synthetic dataset improve evidence citation and summary generation from long contexts by extracting diverse and consistent evidence.
paper_ai_keywords: large language models LLMs query focused summarization unstructured evidence citation Summaries with Unstructured Evidence Text dataset SUnsET positional biases evidence citation document types lengths relevance consistency