title: MLRC-Bench: Can Language Agents Solve Machine Learning Research
    Challenges?
summary: Existing evaluation of large language model (LLM) agents on scientific
  discovery lacks objective baselines and metrics to assess the viability of
  their proposed methods. To address this issue, we introduce MLRC-Bench, a
  benchmark designed to quantify how effectively language agents can tackle
  challenging Machine Learning (ML) Research Competitions. Our benchmark
  highlights open research problems that demand novel methodologies, in contrast
  to recent benchmarks such as OpenAI's MLE-Bench (Chan et al., 2024) and METR's
  RE-Bench (Wijk et al., 2024), which focus on well-established research tasks
  that are largely solvable through sufficient engineering effort. Unlike prior
  work, e.g., AI Scientist (Lu et al., 2024b), which evaluates the end-to-end
  agentic pipeline by using LLM-as-a-judge, MLRC-Bench measures the key steps of
  proposing and implementing novel research methods and evaluates them with newly
  proposed rigorous protocol and objective metrics. Our curated suite of 7
  competition tasks reveals significant challenges for LLM agents. Even the
  best-performing tested agent (gemini-exp-1206 under MLAB (Huang et al., 2024a))
  closes only 9.3% of the gap between baseline and top human participant scores.
  Furthermore, our analysis reveals a misalignment between the LLM-judged
  innovation and their actual performance on cutting-edge ML research problems.
  MLRC-Bench is a dynamic benchmark, which is designed to continually grow with
  new ML competitions to encourage rigorous and objective evaluations of AI's
  research capabilities.
paper_authors._id: 6801913053f9bb6c0d90f07a
paper_authors.name: Yunxiang Zhang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-04-17T23:41:32.260Z
paper_authors._id: 64b7d9f1aa992c4ee9961012
paper_authors.avatarUrl: /avatars/3f818c51d71c43cae8cca815bdb8225f.svg
paper_authors.fullname: Yunxiang Zhang
paper_authors.type: user
paper_authors.user: yunx-z
paper_authors._id: 6801913053f9bb6c0d90f07b
paper_authors.name: Muhammad Khalifa
paper_authors._id: 6801913053f9bb6c0d90f07c
paper_authors.name: Shitanshu Bhushan
paper_authors._id: 6801913053f9bb6c0d90f07d
paper_authors.name: Grant D Murphy
paper_authors._id: 6801913053f9bb6c0d90f07e
paper_authors.name: Lajanugen Logeswaran
paper_authors._id: 6801913053f9bb6c0d90f07f
paper_authors.name: Jaekyeom Kim
paper_authors._id: 6801913053f9bb6c0d90f080
paper_authors.name: Moontae Lee
paper_authors._id: 6801913053f9bb6c0d90f081
paper_authors.name: Honglak Lee
paper_authors._id: 6801913053f9bb6c0d90f082
paper_authors.name: Lu Wang
paper_title: MLRC-Bench: Can Language Agents Solve Machine Learning Research
    Challenges?
paper_summary: Existing evaluation of large language model (LLM) agents on scientific
  discovery lacks objective baselines and metrics to assess the viability of
  their proposed methods. To address this issue, we introduce MLRC-Bench, a
  benchmark designed to quantify how effectively language agents can tackle
  challenging Machine Learning (ML) Research Competitions. Our benchmark
  highlights open research problems that demand novel methodologies, in contrast
  to recent benchmarks such as OpenAI's MLE-Bench (Chan et al., 2024) and METR's
  RE-Bench (Wijk et al., 2024), which focus on well-established research tasks
  that are largely solvable through sufficient engineering effort. Unlike prior
  work, e.g., AI Scientist (Lu et al., 2024b), which evaluates the end-to-end
  agentic pipeline by using LLM-as-a-judge, MLRC-Bench measures the key steps of
  proposing and implementing novel research methods and evaluates them with newly
  proposed rigorous protocol and objective metrics. Our curated suite of 7
  competition tasks reveals significant challenges for LLM agents. Even the
  best-performing tested agent (gemini-exp-1206 under MLAB (Huang et al., 2024a))
  closes only 9.3% of the gap between baseline and top human participant scores.
  Furthermore, our analysis reveals a misalignment between the LLM-judged
  innovation and their actual performance on cutting-edge ML research problems.
  MLRC-Bench is a dynamic benchmark, which is designed to continually grow with
  new ML competitions to encourage rigorous and objective evaluations of AI's
  research capabilities.
paper_ai_summary: MLRC-Bench evaluates large language model agents in tackling novel machine learning research competitions using rigorous protocols and objective metrics, highlighting significant challenges and misalignments compared to previous benchmarks.
paper_ai_keywords: MLRC-Bench machine learning research competitions large language model agents MLE-Bench RE-Bench AI Scientist LLM-as-a-judge MLAB benchmarks objective metrics dynamic benchmark