MLRC-Bench: Can Language Agents Solve Machine Learning Research
  Challenges?
Existing evaluation of large language model (LLM) agents on scientific
discovery lacks objective baselines and metrics to assess the viability of
their proposed methods. To address this issue, we introduce MLRC-Bench, a
benchmark designed to quantify how effectively language agents can tackle
challenging Machine Learning (ML) Research Competitions. Our benchmark
highlights open research problems that demand novel methodologies, in contrast
to recent benchmarks such as OpenAI's MLE-Bench (Chan et al., 2024) and METR's
RE-Bench (Wijk et al., 2024), which focus on well-established research tasks
that are largely solvable through sufficient engineering effort. Unlike prior
work, e.g., AI Scientist (Lu et al., 2024b), which evaluates the end-to-end
agentic pipeline by using LLM-as-a-judge, MLRC-Bench measures the key steps of
proposing and implementing novel research methods and evaluates them with newly
proposed rigorous protocol and objective metrics. Our curated suite of 7
competition tasks reveals significant challenges for LLM agents. Even the
best-performing tested agent (gemini-exp-1206 under MLAB (Huang et al., 2024a))
closes only 9.3% of the gap between baseline and top human participant scores.
Furthermore, our analysis reveals a misalignment between the LLM-judged
innovation and their actual performance on cutting-edge ML research problems.
MLRC-Bench is a dynamic benchmark, which is designed to continually grow with
new ML competitions to encourage rigorous and objective evaluations of AI's
research capabilities.
6801913053f9bb6c0d90f07a
Yunxiang Zhang
extracted_confirmed
2025-04-17T23:41:32.260Z
64b7d9f1aa992c4ee9961012
/avatars/3f818c51d71c43cae8cca815bdb8225f.svg
Yunxiang Zhang
user
yunx-z
6801913053f9bb6c0d90f07b
Muhammad Khalifa
6801913053f9bb6c0d90f07c
Shitanshu Bhushan
6801913053f9bb6c0d90f07d
Grant D Murphy
6801913053f9bb6c0d90f07e
Lajanugen Logeswaran
6801913053f9bb6c0d90f07f
Jaekyeom Kim
6801913053f9bb6c0d90f080
Moontae Lee
6801913053f9bb6c0d90f081
Honglak Lee
6801913053f9bb6c0d90f082
Lu Wang
MLRC-Bench: Can Language Agents Solve Machine Learning Research
  Challenges?
Existing evaluation of large language model (LLM) agents on scientific
discovery lacks objective baselines and metrics to assess the viability of
their proposed methods. To address this issue, we introduce MLRC-Bench, a
benchmark designed to quantify how effectively language agents can tackle
challenging Machine Learning (ML) Research Competitions. Our benchmark
highlights open research problems that demand novel methodologies, in contrast
to recent benchmarks such as OpenAI's MLE-Bench (Chan et al., 2024) and METR's
RE-Bench (Wijk et al., 2024), which focus on well-established research tasks
that are largely solvable through sufficient engineering effort. Unlike prior
work, e.g., AI Scientist (Lu et al., 2024b), which evaluates the end-to-end
agentic pipeline by using LLM-as-a-judge, MLRC-Bench measures the key steps of
proposing and implementing novel research methods and evaluates them with newly
proposed rigorous protocol and objective metrics. Our curated suite of 7
competition tasks reveals significant challenges for LLM agents. Even the
best-performing tested agent (gemini-exp-1206 under MLAB (Huang et al., 2024a))
closes only 9.3% of the gap between baseline and top human participant scores.
Furthermore, our analysis reveals a misalignment between the LLM-judged
innovation and their actual performance on cutting-edge ML research problems.
MLRC-Bench is a dynamic benchmark, which is designed to continually grow with
new ML competitions to encourage rigorous and objective evaluations of AI's
research capabilities.
MLRC-Bench evaluates large language model agents in tackling novel machine learning research competitions using rigorous protocols and objective metrics, highlighting significant challenges and misalignments compared to previous benchmarks.
MLRC-Bench machine learning research competitions large language model agents MLE-Bench RE-Bench AI Scientist LLM-as-a-judge MLAB benchmarks objective metrics dynamic benchmark