Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems
LLMs and RAG systems are now capable of handling millions of input tokens or
more. However, evaluating the output quality of such systems on long-context
tasks remains challenging, as tasks like Needle-in-a-Haystack lack complexity.
In this work, we argue that summarization can play a central role in such
evaluation. We design a procedure to synthesize Haystacks of documents,
ensuring that specific insights repeat across documents. The "Summary
of a Haystack" (SummHay) task then requires a system to process the Haystack
and generate, given a query, a summary that identifies the relevant insights
and precisely cites the source documents. Since we have precise knowledge of
what insights should appear in a haystack summary and what documents should be
cited, we implement a highly reproducible automatic evaluation that can score
summaries on two aspects - Coverage and Citation. We generate Haystacks in two
domains (conversation, news), and perform a large-scale evaluation of 10 LLMs
and corresponding 50 RAG systems. Our findings indicate that SummHay is an open
challenge for current systems, as even systems provided with an Oracle signal
of document relevance lag our estimate of human performance (56\%) by 10+
points on a Joint Score. Without a retriever, long-context LLMs like GPT-4o and
Claude 3 Opus score below 20% on SummHay. We show SummHay can also be used to
study enterprise RAG systems and position bias in long-context models. We hope
future systems can equal and surpass human performance on SummHay.
668504fa45d8ceb44608dff2
Philippe Laban
admin_assigned
2024-07-03T08:46:00.151Z
610aee2dc30263ce69e0dd9c
/avatars/b6e5bba78425e9c28d1b75ee8a7eaac1.svg
Philippe Laban
user
philippelaban
668504fa45d8ceb44608dff3
Alexander R. Fabbri
admin_assigned
2024-07-03T08:46:27.630Z
626427701ed8d81e47ab6899
/avatars/7c89167744703771e3f163665cdb66cc.svg
Alexander Fabbri
user
alexfabbri
668504fa45d8ceb44608dff4
Caiming Xiong
admin_assigned
2024-07-03T08:46:34.196Z
649dbcc4e0fff1ed099dc80a
/avatars/c87c273ca628dbcddccbf1ee19b2ce33.svg
Caiming Xiong
user
cxiong
668504fa45d8ceb44608dff5
Chien-Sheng Wu
admin_assigned
2024-07-03T08:46:48.026Z
5f09073719cb630495b81428
/avatars/e49f53128b5305dfec44d80129c59ca3.svg
Chien-Sheng (Jason) Wu
user
jasonwu
Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems
LLMs and RAG systems are now capable of handling millions of input tokens or
more. However, evaluating the output quality of such systems on long-context
tasks remains challenging, as tasks like Needle-in-a-Haystack lack complexity.
In this work, we argue that summarization can play a central role in such
evaluation. We design a procedure to synthesize Haystacks of documents,
ensuring that specific insights repeat across documents. The "Summary
of a Haystack" (SummHay) task then requires a system to process the Haystack
and generate, given a query, a summary that identifies the relevant insights
and precisely cites the source documents. Since we have precise knowledge of
what insights should appear in a haystack summary and what documents should be
cited, we implement a highly reproducible automatic evaluation that can score
summaries on two aspects - Coverage and Citation. We generate Haystacks in two
domains (conversation, news), and perform a large-scale evaluation of 10 LLMs
and corresponding 50 RAG systems. Our findings indicate that SummHay is an open
challenge for current systems, as even systems provided with an Oracle signal
of document relevance lag our estimate of human performance (56\%) by 10+
points on a Joint Score. Without a retriever, long-context LLMs like GPT-4o and
Claude 3 Opus score below 20% on SummHay. We show SummHay can also be used to
study enterprise RAG systems and position bias in long-context models. We hope
future systems can equal and surpass human performance on SummHay.
The SummHay task evaluates LLMs and RAG systems on summarizing long-context documents by identifying relevant insights and citing sources, revealing challenges even for systems with document relevance signals.
LLMs RAG systems input tokens Needle-in-a-Haystack summarization Haystacks insights Summary of a Haystack Coverage Citation automatic evaluation Conversation News GPT-4o Claude 3 Opus Joint Score Oracle signal long-context models enterprise RAG systems position bias