title: MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research
summary: Recent advancements in AI agents have demonstrated their growing potential to
  drive and support scientific discovery. In this work, we introduce MLR-Bench, a
  comprehensive benchmark for evaluating AI agents on open-ended machine learning
  research. MLR-Bench includes three key components: (1) 201 research tasks
  sourced from NeurIPS, ICLR, and ICML workshops covering diverse ML topics; (2)
  MLR-Judge, an automated evaluation framework combining LLM-based reviewers with
  carefully designed review rubrics to assess research quality; and (3)
  MLR-Agent, a modular agent scaffold capable of completing research tasks
  through four stages: idea generation, proposal formulation, experimentation,
  and paper writing. Our framework supports both stepwise assessment across these
  distinct research stages, and end-to-end evaluation of the final research
  paper. We then use MLR-Bench to evaluate six frontier LLMs and an advanced
  coding agent, finding that while LLMs are effective at generating coherent
  ideas and well-structured papers, current coding agents frequently (e.g., in
  80% of the cases) produce fabricated or invalidated experimental
  results--posing a major barrier to scientific reliability. We validate
  MLR-Judge through human evaluation, showing high agreement with expert
  reviewers, supporting its potential as a scalable tool for research evaluation.
  We open-source MLR-Bench to help the community benchmark, diagnose, and improve
  AI research agents toward trustworthy and transparent scientific discovery.
paper_authors._id: 68354470650d51732c992a4e
paper_authors.name: Hui Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T08:47:12.560Z
paper_authors._id: 61166c4328c98bfd5b92e7c5
paper_authors.avatarUrl: /avatars/f4bb0f0cc2c5b84428c28bddaa479b61.svg
paper_authors.fullname: Hui Chen
paper_authors.type: user
paper_authors.user: chchenhui
paper_authors._id: 68354470650d51732c992a4f
paper_authors.name: Miao Xiong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T10:03:35.714Z
paper_authors._id: 6530cf34e7535baecd9620a7
paper_authors.avatarUrl: /avatars/e6058a932d88e42b4957734f653cbcfd.svg
paper_authors.fullname: Miao Xiong
paper_authors.type: user
paper_authors.user: happymio
paper_authors._id: 68354470650d51732c992a50
paper_authors.name: Yujie Lu
paper_authors._id: 68354470650d51732c992a51
paper_authors.name: Wei Han
paper_authors._id: 68354470650d51732c992a52
paper_authors.name: Ailin Deng
paper_authors._id: 68354470650d51732c992a53
paper_authors.name: Yufei He
paper_authors._id: 68354470650d51732c992a54
paper_authors.name: Jiaying Wu
paper_authors._id: 68354470650d51732c992a55
paper_authors.name: Yibo Li
paper_authors._id: 68354470650d51732c992a56
paper_authors.name: Yue Liu
paper_authors._id: 68354470650d51732c992a57
paper_authors.name: Bryan Hooi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T12:53:11.481Z
paper_authors._id: 651d8032c50012d33e914f2f
paper_authors.avatarUrl: /avatars/0a44c9f51fc50ce86582e328c361ea00.svg
paper_authors.fullname: Bryan Hooi
paper_authors.type: user
paper_authors.user: bhooi
paper_title: MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research
paper_summary: Recent advancements in AI agents have demonstrated their growing potential to
  drive and support scientific discovery. In this work, we introduce MLR-Bench, a
  comprehensive benchmark for evaluating AI agents on open-ended machine learning
  research. MLR-Bench includes three key components: (1) 201 research tasks
  sourced from NeurIPS, ICLR, and ICML workshops covering diverse ML topics; (2)
  MLR-Judge, an automated evaluation framework combining LLM-based reviewers with
  carefully designed review rubrics to assess research quality; and (3)
  MLR-Agent, a modular agent scaffold capable of completing research tasks
  through four stages: idea generation, proposal formulation, experimentation,
  and paper writing. Our framework supports both stepwise assessment across these
  distinct research stages, and end-to-end evaluation of the final research
  paper. We then use MLR-Bench to evaluate six frontier LLMs and an advanced
  coding agent, finding that while LLMs are effective at generating coherent
  ideas and well-structured papers, current coding agents frequently (e.g., in
  80% of the cases) produce fabricated or invalidated experimental
  results--posing a major barrier to scientific reliability. We validate
  MLR-Judge through human evaluation, showing high agreement with expert
  reviewers, supporting its potential as a scalable tool for research evaluation.
  We open-source MLR-Bench to help the community benchmark, diagnose, and improve
  AI research agents toward trustworthy and transparent scientific discovery.
paper_ai_summary: MLR-Bench evaluates AI agents in scientific research through modular stages, revealing that while LLMs perform well in ideation and writing, coding agents often produce unreliable experimental results.
paper_ai_keywords: MLR-Bench MLR-Judge MLR-Agent LLM-based reviewers review rubrics research evaluation idea generation proposal formulation experimentation paper writing