MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research
Recent advancements in AI agents have demonstrated their growing potential to
drive and support scientific discovery. In this work, we introduce MLR-Bench, a
comprehensive benchmark for evaluating AI agents on open-ended machine learning
research. MLR-Bench includes three key components: (1) 201 research tasks
sourced from NeurIPS, ICLR, and ICML workshops covering diverse ML topics; (2)
MLR-Judge, an automated evaluation framework combining LLM-based reviewers with
carefully designed review rubrics to assess research quality; and (3)
MLR-Agent, a modular agent scaffold capable of completing research tasks
through four stages: idea generation, proposal formulation, experimentation,
and paper writing. Our framework supports both stepwise assessment across these
distinct research stages, and end-to-end evaluation of the final research
paper. We then use MLR-Bench to evaluate six frontier LLMs and an advanced
coding agent, finding that while LLMs are effective at generating coherent
ideas and well-structured papers, current coding agents frequently (e.g., in
80% of the cases) produce fabricated or invalidated experimental
results--posing a major barrier to scientific reliability. We validate
MLR-Judge through human evaluation, showing high agreement with expert
reviewers, supporting its potential as a scalable tool for research evaluation.
We open-source MLR-Bench to help the community benchmark, diagnose, and improve
AI research agents toward trustworthy and transparent scientific discovery.
68354470650d51732c992a4e
Hui Chen
claimed_verified
2025-05-27T08:47:12.560Z
61166c4328c98bfd5b92e7c5
/avatars/f4bb0f0cc2c5b84428c28bddaa479b61.svg
Hui Chen
user
chchenhui
68354470650d51732c992a4f
Miao Xiong
claimed_verified
2025-05-27T10:03:35.714Z
6530cf34e7535baecd9620a7
/avatars/e6058a932d88e42b4957734f653cbcfd.svg
Miao Xiong
user
happymio
68354470650d51732c992a50
Yujie Lu
68354470650d51732c992a51
Wei Han
68354470650d51732c992a52
Ailin Deng
68354470650d51732c992a53
Yufei He
68354470650d51732c992a54
Jiaying Wu
68354470650d51732c992a55
Yibo Li
68354470650d51732c992a56
Yue Liu
68354470650d51732c992a57
Bryan Hooi
admin_assigned
2025-05-27T12:53:11.481Z
651d8032c50012d33e914f2f
/avatars/0a44c9f51fc50ce86582e328c361ea00.svg
Bryan Hooi
user
bhooi
MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research
Recent advancements in AI agents have demonstrated their growing potential to
drive and support scientific discovery. In this work, we introduce MLR-Bench, a
comprehensive benchmark for evaluating AI agents on open-ended machine learning
research. MLR-Bench includes three key components: (1) 201 research tasks
sourced from NeurIPS, ICLR, and ICML workshops covering diverse ML topics; (2)
MLR-Judge, an automated evaluation framework combining LLM-based reviewers with
carefully designed review rubrics to assess research quality; and (3)
MLR-Agent, a modular agent scaffold capable of completing research tasks
through four stages: idea generation, proposal formulation, experimentation,
and paper writing. Our framework supports both stepwise assessment across these
distinct research stages, and end-to-end evaluation of the final research
paper. We then use MLR-Bench to evaluate six frontier LLMs and an advanced
coding agent, finding that while LLMs are effective at generating coherent
ideas and well-structured papers, current coding agents frequently (e.g., in
80% of the cases) produce fabricated or invalidated experimental
results--posing a major barrier to scientific reliability. We validate
MLR-Judge through human evaluation, showing high agreement with expert
reviewers, supporting its potential as a scalable tool for research evaluation.
We open-source MLR-Bench to help the community benchmark, diagnose, and improve
AI research agents toward trustworthy and transparent scientific discovery.
MLR-Bench evaluates AI agents in scientific research through modular stages, revealing that while LLMs perform well in ideation and writing, coding agents often produce unreliable experimental results.
MLR-Bench MLR-Judge MLR-Agent LLM-based reviewers review rubrics research evaluation idea generation proposal formulation experimentation paper writing