LongGenBench: Long-context Generation Benchmark
Current long-context benchmarks primarily focus on retrieval-based tests,
requiring Large Language Models (LLMs) to locate specific information within
extensive input contexts, such as the needle-in-a-haystack (NIAH) benchmark.
Long-context generation refers to the ability of a language model to generate
coherent and contextually accurate text that spans across lengthy passages or
documents. While recent studies show strong performance on NIAH and other
retrieval-based long-context benchmarks, there is a significant lack of
benchmarks for evaluating long-context generation capabilities. To bridge this
gap and offer a comprehensive assessment, we introduce a synthetic benchmark,
LongGenBench, which allows for flexible configurations of customized generation
context lengths. LongGenBench advances beyond traditional benchmarks by
redesigning the format of questions and necessitating that LLMs respond with a
single, cohesive long-context answer. Upon extensive evaluation using
LongGenBench, we observe that: (1) both API accessed and open source models
exhibit performance degradation in long-context generation scenarios, ranging
from 1.2% to 47.1%; (2) different series of LLMs exhibit varying trends of
performance degradation, with the Gemini-1.5-Flash model showing the least
degradation among API accessed models, and the Qwen2 series exhibiting the
least degradation in LongGenBench among open source models.
6705ecde7583ffc1e02f01f8
Xiang Liu
claimed_verified
2024-10-09T07:37:24.402Z
63024676056ec3a2a8714b24
https://cdn-avatars.huggingface.co/v1/production/uploads/1661093436322-noauth.jpeg
Xiang Liu
user
Dominic789654
6705ecde7583ffc1e02f01f9
Peijie Dong
claimed_verified
2025-06-02T07:57:16.473Z
6395f845aec00abff778ad31
https://cdn-avatars.huggingface.co/v1/production/uploads/6395f845aec00abff778ad31/bZkAlchSvqER1HgBKmcHI.jpeg
PeijieDong
user
pprp
6705ecde7583ffc1e02f01fa
Xuming Hu
admin_assigned
2024-10-09T08:59:09.897Z
66545bbcc2e9c65dcc1ee57b
/avatars/120834f04f86e2f7be33c9eb6d99b1fa.svg
minghui
user
xuminghui
6705ecde7583ffc1e02f01fb
Xiaowen Chu
admin_assigned
2024-10-09T08:59:03.917Z
6676935fcd0b89a0115174b0
/avatars/4caca1b672d29e787814f9a30bf20bcc.svg
Xiaowen Chu
user
wenxinsiju
LongGenBench: Long-context Generation Benchmark
Current long-context benchmarks primarily focus on retrieval-based tests,
requiring Large Language Models (LLMs) to locate specific information within
extensive input contexts, such as the needle-in-a-haystack (NIAH) benchmark.
Long-context generation refers to the ability of a language model to generate
coherent and contextually accurate text that spans across lengthy passages or
documents. While recent studies show strong performance on NIAH and other
retrieval-based long-context benchmarks, there is a significant lack of
benchmarks for evaluating long-context generation capabilities. To bridge this
gap and offer a comprehensive assessment, we introduce a synthetic benchmark,
LongGenBench, which allows for flexible configurations of customized generation
context lengths. LongGenBench advances beyond traditional benchmarks by
redesigning the format of questions and necessitating that LLMs respond with a
single, cohesive long-context answer. Upon extensive evaluation using
LongGenBench, we observe that: (1) both API accessed and open source models
exhibit performance degradation in long-context generation scenarios, ranging
from 1.2% to 47.1%; (2) different series of LLMs exhibit varying trends of
performance degradation, with the Gemini-1.5-Flash model showing the least
degradation among API accessed models, and the Qwen2 series exhibiting the
least degradation in LongGenBench among open source models.
A new benchmark, LongGenBench, evaluates the long-context generation capabilities of large language models, revealing varying performance degradation across different models and model series.
Large Language Models (LLMs) long-context generation needle-in-a-haystack (NIAH) LongGenBench synthetic benchmark context lengths generation context performance degradation Gemini-1.5-Flash Qwen2