Unstructured Evidence Attribution for Long Context Query Focused
  Summarization
Large language models (LLMs) are capable of generating coherent summaries
from very long contexts given a user query. Extracting and properly citing
evidence spans could help improve the transparency and reliability of these
summaries. At the same time, LLMs suffer from positional biases in terms of
which information they understand and attend to, which could affect evidence
citation. Whereas previous work has focused on evidence citation with
predefined levels of granularity (e.g. sentence, paragraph, document, etc.), we
propose the task of long-context query focused summarization with unstructured
evidence citation. We show how existing systems struggle to generate and
properly cite unstructured evidence from their context, and that evidence tends
to be "lost-in-the-middle". To help mitigate this, we create the Summaries with
Unstructured Evidence Text dataset (SUnsET), a synthetic dataset generated
using a novel domain-agnostic pipeline which can be used as supervision to
adapt LLMs to this task. We demonstrate across 5 LLMs of different sizes and 4
datasets with varying document types and lengths that LLMs adapted with SUnsET
data generate more relevant and factually consistent evidence than their base
models, extract evidence from more diverse locations in their context, and can
generate more relevant and consistent summaries.
67b83a20a9fa331061e84ecd
Dustin Wright
claimed_verified
2025-02-21T09:58:02.288Z
60a643b9213fe60589b8fdf9
https://cdn-avatars.huggingface.co/v1/production/uploads/60a643b9213fe60589b8fdf9/OOXmW3MkSf88r63tAE6-n.jpeg
Dustin Wright
user
dwright37
67b83a20a9fa331061e84ece
Zain Muhammad Mujahid
admin_assigned
2025-02-21T15:16:52.600Z
637e8b1b66ee00bcb2468ed0
https://cdn-avatars.huggingface.co/v1/production/uploads/1669240174964-637e8b1b66ee00bcb2468ed0.jpeg
Zain
user
zainmujahid
67b83a20a9fa331061e84ecf
Lu Wang
67b83a20a9fa331061e84ed0
Isabelle Augenstein
admin_assigned
2025-02-21T15:17:02.420Z
608918b7df398c3b285ce960
https://cdn-avatars.huggingface.co/v1/production/uploads/1621507769190-608918b7df398c3b285ce960.jpeg
Isabelle Augenstein
user
IAugenstein
67b83a20a9fa331061e84ed1
David Jurgens
admin_assigned
2025-02-21T15:17:08.686Z
63516acdce7cf1fe8a854cdc
https://cdn-avatars.huggingface.co/v1/production/uploads/63516acdce7cf1fe8a854cdc/TlOI7iPdG7zJKWyGLoPQN.jpeg
David Jurgens
user
davidjurgens
Unstructured Evidence Attribution for Long Context Query Focused
  Summarization
Large language models (LLMs) are capable of generating coherent summaries
from very long contexts given a user query. Extracting and properly citing
evidence spans could help improve the transparency and reliability of these
summaries. At the same time, LLMs suffer from positional biases in terms of
which information they understand and attend to, which could affect evidence
citation. Whereas previous work has focused on evidence citation with
predefined levels of granularity (e.g. sentence, paragraph, document, etc.), we
propose the task of long-context query focused summarization with unstructured
evidence citation. We show how existing systems struggle to generate and
properly cite unstructured evidence from their context, and that evidence tends
to be "lost-in-the-middle". To help mitigate this, we create the Summaries with
Unstructured Evidence Text dataset (SUnsET), a synthetic dataset generated
using a novel domain-agnostic pipeline which can be used as supervision to
adapt LLMs to this task. We demonstrate across 5 LLMs of different sizes and 4
datasets with varying document types and lengths that LLMs adapted with SUnsET
data generate more relevant and factually consistent evidence than their base
models, extract evidence from more diverse locations in their context, and can
generate more relevant and consistent summaries.
LMMs adapted with a new synthetic dataset improve evidence citation and summary generation from long contexts by extracting diverse and consistent evidence.
large language models LLMs query focused summarization unstructured evidence citation Summaries with Unstructured Evidence Text dataset SUnsET positional biases evidence citation document types lengths relevance consistency