title: A Controllable Examination for Long-Context Language Models
summary: Existing frameworks for evaluating long-context language models (LCLM) can be
  broadly categorized into real-world and synthetic tasks. Despite their utility,
  both approaches are accompanied by certain intrinsic limitations. Real-world
  tasks are too complex to interpret or characterize and are susceptible to data
  contamination. In contrast, synthetic tasks often adopt the
  needle-in-the-haystack (NIAH) format, wherein a lack of coherence between the
  "needle" and the "haystack" compromises their validity as proxies for realistic
  applications. In response to these challenges, we posit that an ideal
  long-context evaluation framework should be characterized by three essential
  features: seamless context, controllable setting, and
  sound evaluation. This study introduces LongBioBench, a
  novel benchmark that utilizes artificially generated biographies as a
  controlled environment for assessing LCLMs across dimensions of
  understanding, reasoning, and trustworthiness.
  Our experimental evaluation, which includes 18 LCLMs in total,
  demonstrates that most models still exhibit deficiencies in semantic
  understanding and elementary reasoning over retrieved results and are less
  trustworthy as context length increases. Our further analysis indicates some
  design choices employed by existing synthetic benchmarks, such as contextual
  non-coherence, numerical needles, and the absence of distractors, rendering
  them vulnerable to test the model long-context capabilities. Moreover, we also
  reveal that long-context continual pretraining primarily adjusts RoPE embedding
  to accommodate extended context lengths. To sum up, compared to previous
  synthetic benchmarks, LongBioBench achieves a better trade-off between
  mirroring authentic language tasks and maintaining controllability, and is
  highly interpretable and configurable.
paper_authors._id: 683ff4dcfbc9041ef7274c51
paper_authors.name: Yijun Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T08:53:47.455Z
paper_authors._id: 657eea68f4f72f2c4c44640d
paper_authors.avatarUrl: /avatars/033bc4f063cd36a79a0b4761f6ebe32c.svg
paper_authors.fullname: Yijun YANG
paper_authors.type: user
paper_authors.user: thomasyyj
paper_authors._id: 683ff4dcfbc9041ef7274c52
paper_authors.name: Zeyu Huang
paper_authors._id: 683ff4dcfbc9041ef7274c53
paper_authors.name: Wenhao Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T12:47:37.569Z
paper_authors._id: 649d7d8968586ca9bf7f5fe6
paper_authors.avatarUrl: /avatars/b444240770d4025dea41871cf38126dc.svg
paper_authors.fullname: Wenhao Zhu
paper_authors.type: user
paper_authors.user: Wenhao97
paper_authors._id: 683ff4dcfbc9041ef7274c54
paper_authors.name: Zihan Qiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T12:47:15.684Z
paper_authors._id: 647ccbd6e07cf9bb2d485244
paper_authors.avatarUrl: /avatars/e8915abaff04f6762247e196b7cf84df.svg
paper_authors.fullname: Zihan Qiu
paper_authors.type: user
paper_authors.user: QwQZh
paper_authors._id: 683ff4dcfbc9041ef7274c55
paper_authors.name: Fei Yuan
paper_authors._id: 683ff4dcfbc9041ef7274c56
paper_authors.name: Jeff Z. Pan
paper_authors._id: 683ff4dcfbc9041ef7274c57
paper_authors.name: Ivan Titov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T12:46:49.924Z
paper_authors._id: 666f6375623133f1ce79c021
paper_authors.avatarUrl: /avatars/b57a457cff81d685fdd9eadee53445fc.svg
paper_authors.fullname: Ivan Titov
paper_authors.type: user
paper_authors.user: Ivanchoo
paper_title: A Controllable Examination for Long-Context Language Models
paper_summary: Existing frameworks for evaluating long-context language models (LCLM) can be
  broadly categorized into real-world and synthetic tasks. Despite their utility,
  both approaches are accompanied by certain intrinsic limitations. Real-world
  tasks are too complex to interpret or characterize and are susceptible to data
  contamination. In contrast, synthetic tasks often adopt the
  needle-in-the-haystack (NIAH) format, wherein a lack of coherence between the
  "needle" and the "haystack" compromises their validity as proxies for realistic
  applications. In response to these challenges, we posit that an ideal
  long-context evaluation framework should be characterized by three essential
  features: seamless context, controllable setting, and
  sound evaluation. This study introduces LongBioBench, a
  novel benchmark that utilizes artificially generated biographies as a
  controlled environment for assessing LCLMs across dimensions of
  understanding, reasoning, and trustworthiness.
  Our experimental evaluation, which includes 18 LCLMs in total,
  demonstrates that most models still exhibit deficiencies in semantic
  understanding and elementary reasoning over retrieved results and are less
  trustworthy as context length increases. Our further analysis indicates some
  design choices employed by existing synthetic benchmarks, such as contextual
  non-coherence, numerical needles, and the absence of distractors, rendering
  them vulnerable to test the model long-context capabilities. Moreover, we also
  reveal that long-context continual pretraining primarily adjusts RoPE embedding
  to accommodate extended context lengths. To sum up, compared to previous
  synthetic benchmarks, LongBioBench achieves a better trade-off between
  mirroring authentic language tasks and maintaining controllability, and is
  highly interpretable and configurable.
paper_ai_summary: LongBioBench is a new benchmark using artificially generated biographies to evaluate long-context language models across understanding, reasoning, and trustworthiness dimensions, addressing limitations in existing frameworks.
paper_ai_keywords: long-context language models (LCLM) real-world tasks synthetic tasks needle-in-the-haystack (NIAH) seamless context controllable setting sound evaluation LongBioBench semantic understanding elementary reasoning trustworthiness long-context continual pretraining RoPE embedding