title: Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems
summary: LLMs and RAG systems are now capable of handling millions of input tokens or
  more. However, evaluating the output quality of such systems on long-context
  tasks remains challenging, as tasks like Needle-in-a-Haystack lack complexity.
  In this work, we argue that summarization can play a central role in such
  evaluation. We design a procedure to synthesize Haystacks of documents,
  ensuring that specific insights repeat across documents. The "Summary
  of a Haystack" (SummHay) task then requires a system to process the Haystack
  and generate, given a query, a summary that identifies the relevant insights
  and precisely cites the source documents. Since we have precise knowledge of
  what insights should appear in a haystack summary and what documents should be
  cited, we implement a highly reproducible automatic evaluation that can score
  summaries on two aspects - Coverage and Citation. We generate Haystacks in two
  domains (conversation, news), and perform a large-scale evaluation of 10 LLMs
  and corresponding 50 RAG systems. Our findings indicate that SummHay is an open
  challenge for current systems, as even systems provided with an Oracle signal
  of document relevance lag our estimate of human performance (56\%) by 10+
  points on a Joint Score. Without a retriever, long-context LLMs like GPT-4o and
  Claude 3 Opus score below 20% on SummHay. We show SummHay can also be used to
  study enterprise RAG systems and position bias in long-context models. We hope
  future systems can equal and surpass human performance on SummHay.
paper_authors._id: 668504fa45d8ceb44608dff2
paper_authors.name: Philippe Laban
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T08:46:00.151Z
paper_authors._id: 610aee2dc30263ce69e0dd9c
paper_authors.avatarUrl: /avatars/b6e5bba78425e9c28d1b75ee8a7eaac1.svg
paper_authors.fullname: Philippe Laban
paper_authors.type: user
paper_authors.user: philippelaban
paper_authors._id: 668504fa45d8ceb44608dff3
paper_authors.name: Alexander R. Fabbri
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T08:46:27.630Z
paper_authors._id: 626427701ed8d81e47ab6899
paper_authors.avatarUrl: /avatars/7c89167744703771e3f163665cdb66cc.svg
paper_authors.fullname: Alexander Fabbri
paper_authors.type: user
paper_authors.user: alexfabbri
paper_authors._id: 668504fa45d8ceb44608dff4
paper_authors.name: Caiming Xiong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T08:46:34.196Z
paper_authors._id: 649dbcc4e0fff1ed099dc80a
paper_authors.avatarUrl: /avatars/c87c273ca628dbcddccbf1ee19b2ce33.svg
paper_authors.fullname: Caiming Xiong
paper_authors.type: user
paper_authors.user: cxiong
paper_authors._id: 668504fa45d8ceb44608dff5
paper_authors.name: Chien-Sheng Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T08:46:48.026Z
paper_authors._id: 5f09073719cb630495b81428
paper_authors.avatarUrl: /avatars/e49f53128b5305dfec44d80129c59ca3.svg
paper_authors.fullname: Chien-Sheng (Jason) Wu
paper_authors.type: user
paper_authors.user: jasonwu
paper_title: Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems
paper_summary: LLMs and RAG systems are now capable of handling millions of input tokens or
  more. However, evaluating the output quality of such systems on long-context
  tasks remains challenging, as tasks like Needle-in-a-Haystack lack complexity.
  In this work, we argue that summarization can play a central role in such
  evaluation. We design a procedure to synthesize Haystacks of documents,
  ensuring that specific insights repeat across documents. The "Summary
  of a Haystack" (SummHay) task then requires a system to process the Haystack
  and generate, given a query, a summary that identifies the relevant insights
  and precisely cites the source documents. Since we have precise knowledge of
  what insights should appear in a haystack summary and what documents should be
  cited, we implement a highly reproducible automatic evaluation that can score
  summaries on two aspects - Coverage and Citation. We generate Haystacks in two
  domains (conversation, news), and perform a large-scale evaluation of 10 LLMs
  and corresponding 50 RAG systems. Our findings indicate that SummHay is an open
  challenge for current systems, as even systems provided with an Oracle signal
  of document relevance lag our estimate of human performance (56\%) by 10+
  points on a Joint Score. Without a retriever, long-context LLMs like GPT-4o and
  Claude 3 Opus score below 20% on SummHay. We show SummHay can also be used to
  study enterprise RAG systems and position bias in long-context models. We hope
  future systems can equal and surpass human performance on SummHay.
paper_ai_keywords: LLMs (Large Language Models) RAG systems (Retrieval-Augmented Generation systems) Needle-in-a-Haystack tasks summarization Summary of a Haystack (SummHay) task overview generation insights document citations automatic evaluation coverage citation document relevance long-context models GPT-4o Claude 3 Opus enterprise RAG systems position bias