ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form
  Generation Tasks with Structured Checklists
This paper introduces ExpertLongBench, an expert-level benchmark containing
11 tasks from 9 domains that reflect realistic expert workflows and
applications. Beyond question answering, the application-driven tasks in
ExpertLongBench demand long-form outputs that can exceed 5,000 tokens and
strict adherence to domain-specific requirements. Notably, each task in
ExpertLongBench includes a rubric, designed or validated by domain experts, to
specify task requirements and guide output evaluation. Furthermore, we propose
CLEAR, an evaluation framework that supports accurate evaluation of long-form
model outputs in our benchmark. To achieve fine-grained, expert-aligned
evaluation, CLEAR derives checklists from both model outputs and references by
extracting information corresponding to items in the task-specific rubric.
Checklist items for model outputs are then compared with corresponding items
for reference outputs to assess their correctness, enabling grounded
evaluation. We benchmark 11 large language models (LLMs) and analyze components
in CLEAR, showing that (1) existing LLMs, with the top performer achieving only
a 26.8% F1 score, require significant improvement for expert-level tasks; (2)
models can generate content corresponding to the required aspects, though often
not accurately; and (3) accurate checklist extraction and comparison in CLEAR
can be achieved by open-weight models for more scalable and low-cost usage.
68432c693ec10bdd8ab4d75e
Jie Ruan
claimed_verified
2025-06-10T09:22:12.213Z
66d497b3ad293ffc4bebfa60
/avatars/8a5cfd01a9bf9e53803d3d70c5887c16.svg
Jie Ruan
user
JieRuan
68432c693ec10bdd8ab4d75f
Inderjeet Nair
68432c693ec10bdd8ab4d760
Shuyang Cao
68432c693ec10bdd8ab4d761
Amy Liu
claimed_verified
2025-06-11T08:36:52.570Z
64a2ddc6ef638e7f0dc7d996
/avatars/b193c9865dd31c6b27cab8cea15c26ab.svg
Amy Liu
user
amyliiu
68432c693ec10bdd8ab4d762
Sheza Munir
68432c693ec10bdd8ab4d763
Micah Pollens-Dempsey
68432c693ec10bdd8ab4d764
Tiffany Chiang
68432c693ec10bdd8ab4d765
Lucy Kates
68432c693ec10bdd8ab4d766
Nicholas David
68432c693ec10bdd8ab4d767
Sihan Chen
68432c693ec10bdd8ab4d768
Ruxin Yang
68432c693ec10bdd8ab4d769
Yuqian Yang
68432c693ec10bdd8ab4d76a
Jasmine Gump
68432c693ec10bdd8ab4d76b
Tessa Bialek
68432c693ec10bdd8ab4d76c
Vivek Sankaran
68432c693ec10bdd8ab4d76d
Margo Schlanger
68432c693ec10bdd8ab4d76e
Lu Wang
ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form
  Generation Tasks with Structured Checklists
This paper introduces ExpertLongBench, an expert-level benchmark containing
11 tasks from 9 domains that reflect realistic expert workflows and
applications. Beyond question answering, the application-driven tasks in
ExpertLongBench demand long-form outputs that can exceed 5,000 tokens and
strict adherence to domain-specific requirements. Notably, each task in
ExpertLongBench includes a rubric, designed or validated by domain experts, to
specify task requirements and guide output evaluation. Furthermore, we propose
CLEAR, an evaluation framework that supports accurate evaluation of long-form
model outputs in our benchmark. To achieve fine-grained, expert-aligned
evaluation, CLEAR derives checklists from both model outputs and references by
extracting information corresponding to items in the task-specific rubric.
Checklist items for model outputs are then compared with corresponding items
for reference outputs to assess their correctness, enabling grounded
evaluation. We benchmark 11 large language models (LLMs) and analyze components
in CLEAR, showing that (1) existing LLMs, with the top performer achieving only
a 26.8% F1 score, require significant improvement for expert-level tasks; (2)
models can generate content corresponding to the required aspects, though often
not accurately; and (3) accurate checklist extraction and comparison in CLEAR
can be achieved by open-weight models for more scalable and low-cost usage.
ExpertLongBench is a benchmark for expert-level workflows with diverse, long-form tasks, evaluated using CLEAR, a framework that assesses outputs based on domain-specific rubrics and checklists.
ExpertLongBench long-form outputs domain-specific requirements rubric CLEAR evaluation framework checklist large language models (LLMs) F1 score checklist extraction checklist comparison