LongGenBench: Long-context Generation Benchmark
<|reference_start|>LongGenBench: Long-context Generation Benchmark: Current long-context benchmarks primarily focus on retrieval-based tests, requiring Large Language Models (LLMs) to locate specific information within extensive input contexts, such as the needle-in-a-haystack (NIAH) benchmark. Long-context generation refers to the ability of a language model to generate coherent and contextually accurate text that spans across lengthy passages or documents. While recent studies show strong performance on NIAH and other retrieval-based long-context benchmarks, there is a significant lack of benchmarks for evaluating long-context generation capabilities. To bridge this gap and offer a comprehensive assessment, we introduce a synthetic benchmark, LongGenBench, which allows for flexible configurations of customized generation context lengths. LongGenBench advances beyond traditional benchmarks by redesigning the format of questions and necessitating that LLMs respond with a single, cohesive long-context answer. Upon extensive evaluation using LongGenBench, we observe that: (1) both API accessed and open source models exhibit performance degradation in long-context generation scenarios, ranging from 1.2% to 47.1%; (2) different series of LLMs exhibit varying trends of performance degradation, with the Gemini-1.5-Flash model showing the least degradation among API accessed models, and the Qwen2 series exhibiting the least degradation in LongGenBench among open source models.<|reference_end|>
@article{liu2024longgenbench:,
    title={LongGenBench: Long-context Generation Benchmark},
    author={Xiang Liu, Peijie Dong, Xuming Hu, Xiaowen Chu},
    journal={arXiv preprint arXiv:2410.04199},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.04199},
    primaryClass={cs.CL cs.AI}
}