title: LongGenBench: Long-context Generation Benchmark
summary: Current long-context benchmarks primarily focus on retrieval-based tests,
  requiring Large Language Models (LLMs) to locate specific information within
  extensive input contexts, such as the needle-in-a-haystack (NIAH) benchmark.
  Long-context generation refers to the ability of a language model to generate
  coherent and contextually accurate text that spans across lengthy passages or
  documents. While recent studies show strong performance on NIAH and other
  retrieval-based long-context benchmarks, there is a significant lack of
  benchmarks for evaluating long-context generation capabilities. To bridge this
  gap and offer a comprehensive assessment, we introduce a synthetic benchmark,
  LongGenBench, which allows for flexible configurations of customized generation
  context lengths. LongGenBench advances beyond traditional benchmarks by
  redesigning the format of questions and necessitating that LLMs respond with a
  single, cohesive long-context answer. Upon extensive evaluation using
  LongGenBench, we observe that: (1) both API accessed and open source models
  exhibit performance degradation in long-context generation scenarios, ranging
  from 1.2% to 47.1%; (2) different series of LLMs exhibit varying trends of
  performance degradation, with the Gemini-1.5-Flash model showing the least
  degradation among API accessed models, and the Qwen2 series exhibiting the
  least degradation in LongGenBench among open source models.
paper_authors._id: 6705ecde7583ffc1e02f01f8
paper_authors.name: Xiang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-09T07:37:24.402Z
paper_authors._id: 63024676056ec3a2a8714b24
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661093436322-noauth.jpeg
paper_authors.fullname: Xiang Liu
paper_authors.type: user
paper_authors.user: Dominic789654
paper_authors._id: 6705ecde7583ffc1e02f01f9
paper_authors.name: Peijie Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:57:16.473Z
paper_authors._id: 6395f845aec00abff778ad31
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6395f845aec00abff778ad31/bZkAlchSvqER1HgBKmcHI.jpeg
paper_authors.fullname: PeijieDong
paper_authors.type: user
paper_authors.user: pprp
paper_authors._id: 6705ecde7583ffc1e02f01fa
paper_authors.name: Xuming Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-09T08:59:09.897Z
paper_authors._id: 66545bbcc2e9c65dcc1ee57b
paper_authors.avatarUrl: /avatars/120834f04f86e2f7be33c9eb6d99b1fa.svg
paper_authors.fullname: minghui
paper_authors.type: user
paper_authors.user: xuminghui
paper_authors._id: 6705ecde7583ffc1e02f01fb
paper_authors.name: Xiaowen Chu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-09T08:59:03.917Z
paper_authors._id: 6676935fcd0b89a0115174b0
paper_authors.avatarUrl: /avatars/4caca1b672d29e787814f9a30bf20bcc.svg
paper_authors.fullname: Xiaowen Chu
paper_authors.type: user
paper_authors.user: wenxinsiju
paper_title: LongGenBench: Long-context Generation Benchmark
paper_summary: Current long-context benchmarks primarily focus on retrieval-based tests,
  requiring Large Language Models (LLMs) to locate specific information within
  extensive input contexts, such as the needle-in-a-haystack (NIAH) benchmark.
  Long-context generation refers to the ability of a language model to generate
  coherent and contextually accurate text that spans across lengthy passages or
  documents. While recent studies show strong performance on NIAH and other
  retrieval-based long-context benchmarks, there is a significant lack of
  benchmarks for evaluating long-context generation capabilities. To bridge this
  gap and offer a comprehensive assessment, we introduce a synthetic benchmark,
  LongGenBench, which allows for flexible configurations of customized generation
  context lengths. LongGenBench advances beyond traditional benchmarks by
  redesigning the format of questions and necessitating that LLMs respond with a
  single, cohesive long-context answer. Upon extensive evaluation using
  LongGenBench, we observe that: (1) both API accessed and open source models
  exhibit performance degradation in long-context generation scenarios, ranging
  from 1.2% to 47.1%; (2) different series of LLMs exhibit varying trends of
  performance degradation, with the Gemini-1.5-Flash model showing the least
  degradation among API accessed models, and the Qwen2 series exhibiting the
  least degradation in LongGenBench among open source models.
paper_ai_summary: A new benchmark, LongGenBench, evaluates the long-context generation capabilities of large language models, revealing varying performance degradation across different models and model series.
paper_ai_keywords: Large Language Models (LLMs) long-context generation needle-in-a-haystack (NIAH) LongGenBench synthetic benchmark context lengths generation context performance degradation Gemini-1.5-Flash Qwen2