CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models
<|reference_start|>CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models: Developing Large Language Models (LLMs) with robust long-context capabilities has been the recent research focus, resulting in the emergence of long-context LLMs proficient in Chinese. However, the evaluation of these models remains underdeveloped due to a lack of benchmarks. To address this gap, we present CLongEval, a comprehensive Chinese benchmark for evaluating long-context LLMs. CLongEval is characterized by three key features: (1) Sufficient data volume, comprising 7 distinct tasks and 7,267 examples; (2) Broad applicability, accommodating to models with context windows size from 1K to 100K; (3) High quality, with over 2,000 manually annotated question-answer pairs in addition to the automatically constructed labels. With CLongEval, we undertake a comprehensive assessment of 6 open-source long-context LLMs and 2 leading commercial counterparts that feature both long-context abilities and proficiency in Chinese. We also provide in-depth analysis based on the empirical results, trying to shed light on the critical capabilities that present challenges in long-context settings. The dataset, evaluation scripts, and model outputs will be released.<|reference_end|>
@article{qiu2024clongeval:,
    title={CLongEval: A Chinese Benchmark for Evaluating Long-Context Large
  Language Models},
    author={Zexuan Qiu, Jingjing Li, Shijue Huang, Xiaoqi Jiao, Wanjun Zhong,
  Irwin King},
    journal={arXiv preprint arXiv:2403.03514},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.03514},
    primaryClass={cs.CL}
}