AbGen: Evaluating Large Language Models in Ablation Study Design and
  Evaluation for Scientific Research
We introduce AbGen, the first benchmark designed to evaluate the capabilities
of LLMs in designing ablation studies for scientific research. AbGen consists
of 1,500 expert-annotated examples derived from 807 NLP papers. In this
benchmark, LLMs are tasked with generating detailed ablation study designs for
a specified module or process based on the given research context. Our
evaluation of leading LLMs, such as DeepSeek-R1-0528 and o4-mini, highlights a
significant performance gap between these models and human experts in terms of
the importance, faithfulness, and soundness of the ablation study designs.
Moreover, we demonstrate that current automated evaluation methods are not
reliable for our task, as they show a significant discrepancy when compared to
human assessment. To better investigate this, we develop AbGen-Eval, a
meta-evaluation benchmark designed to assess the reliability of commonly used
automated evaluation systems in measuring LLM performance on our task. We
investigate various LLM-as-Judge systems on AbGen-Eval, providing insights for
future research on developing more effective and reliable LLM-based evaluation
systems for complex scientific tasks.
6879c27e21b37e676c8e41a9
Yilun Zhao
6879c27e21b37e676c8e41aa
Weiyuan Chen
6879c27e21b37e676c8e41ab
Zhijian Xu
6879c27e21b37e676c8e41ac
Manasi Patwardhan
6879c27e21b37e676c8e41ad
Yixin Liu
6879c27e21b37e676c8e41ae
Chengye Wang
6879c27e21b37e676c8e41af
Lovekesh Vig
6879c27e21b37e676c8e41b0
Arman Cohan
AbGen: Evaluating Large Language Models in Ablation Study Design and
  Evaluation for Scientific Research
We introduce AbGen, the first benchmark designed to evaluate the capabilities
of LLMs in designing ablation studies for scientific research. AbGen consists
of 1,500 expert-annotated examples derived from 807 NLP papers. In this
benchmark, LLMs are tasked with generating detailed ablation study designs for
a specified module or process based on the given research context. Our
evaluation of leading LLMs, such as DeepSeek-R1-0528 and o4-mini, highlights a
significant performance gap between these models and human experts in terms of
the importance, faithfulness, and soundness of the ablation study designs.
Moreover, we demonstrate that current automated evaluation methods are not
reliable for our task, as they show a significant discrepancy when compared to
human assessment. To better investigate this, we develop AbGen-Eval, a
meta-evaluation benchmark designed to assess the reliability of commonly used
automated evaluation systems in measuring LLM performance on our task. We
investigate various LLM-as-Judge systems on AbGen-Eval, providing insights for
future research on developing more effective and reliable LLM-based evaluation
systems for complex scientific tasks.
AbGen evaluates LLMs in designing ablation studies for scientific research, revealing performance gaps compared to human experts and highlighting the unreliability of current automated evaluation methods.
LLMs ablation studies NLP papers DeepSeek-R1-0528 o4-mini AbGen-Eval LLM-as-Judge