title: EXP-Bench: Can AI Conduct AI Research Experiments?
summary: Automating AI research holds immense potential for accelerating scientific
  progress, yet current AI agents struggle with the complexities of rigorous,
  end-to-end experimentation. We introduce EXP-Bench, a novel benchmark designed
  to systematically evaluate AI agents on complete research experiments sourced
  from influential AI publications. Given a research question and incomplete
  starter code, EXP-Bench challenges AI agents to formulate hypotheses, design
  and implement experimental procedures, execute them, and analyze results. To
  enable the creation of such intricate and authentic tasks with high-fidelity,
  we design a semi-autonomous pipeline to extract and structure crucial
  experimental details from these research papers and their associated
  open-source code. With the pipeline, EXP-Bench curated 461 AI research tasks
  from 51 top-tier AI research papers. Evaluations of leading LLM-based agents,
  such as OpenHands and IterativeAgent on EXP-Bench demonstrate partial
  capabilities: while scores on individual experimental aspects such as design or
  implementation correctness occasionally reach 20-35%, the success rate for
  complete, executable experiments was a mere 0.5%. By identifying these
  bottlenecks and providing realistic step-by-step experiment procedures,
  EXP-Bench serves as a vital tool for future AI agents to improve their ability
  to conduct AI research experiments. EXP-Bench is open-sourced at
  https://github.com/Just-Curieous/Curie/tree/main/benchmark/exp_bench.
paper_authors._id: 683dfe8b4acfa22520c6a9ed
paper_authors.name: Patrick Tser Jern Kon
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T08:47:31.338Z
paper_authors._id: 64b7111e17681d64b19cf95e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b7111e17681d64b19cf95e/VHPfCUl1nBS3OMMVi96CR.jpeg
paper_authors.fullname: Patrick Kon
paper_authors.type: user
paper_authors.user: patkon
paper_authors._id: 683dfe8b4acfa22520c6a9ee
paper_authors.name: Jiachen Liu
paper_authors._id: 683dfe8b4acfa22520c6a9ef
paper_authors.name: Xinyi Zhu
paper_authors._id: 683dfe8b4acfa22520c6a9f0
paper_authors.name: Qiuyi Ding
paper_authors._id: 683dfe8b4acfa22520c6a9f1
paper_authors.name: Jingjia Peng
paper_authors._id: 683dfe8b4acfa22520c6a9f2
paper_authors.name: Jiarong Xing
paper_authors._id: 683dfe8b4acfa22520c6a9f3
paper_authors.name: Yibo Huang
paper_authors._id: 683dfe8b4acfa22520c6a9f4
paper_authors.name: Yiming Qiu
paper_authors._id: 683dfe8b4acfa22520c6a9f5
paper_authors.name: Jayanth Srinivasa
paper_authors._id: 683dfe8b4acfa22520c6a9f6
paper_authors.name: Myungjin Lee
paper_authors._id: 683dfe8b4acfa22520c6a9f7
paper_authors.name: Mosharaf Chowdhury
paper_authors._id: 683dfe8b4acfa22520c6a9f8
paper_authors.name: Matei Zaharia
paper_authors._id: 683dfe8b4acfa22520c6a9f9
paper_authors.name: Ang Chen
paper_title: EXP-Bench: Can AI Conduct AI Research Experiments?
paper_summary: Automating AI research holds immense potential for accelerating scientific
  progress, yet current AI agents struggle with the complexities of rigorous,
  end-to-end experimentation. We introduce EXP-Bench, a novel benchmark designed
  to systematically evaluate AI agents on complete research experiments sourced
  from influential AI publications. Given a research question and incomplete
  starter code, EXP-Bench challenges AI agents to formulate hypotheses, design
  and implement experimental procedures, execute them, and analyze results. To
  enable the creation of such intricate and authentic tasks with high-fidelity,
  we design a semi-autonomous pipeline to extract and structure crucial
  experimental details from these research papers and their associated
  open-source code. With the pipeline, EXP-Bench curated 461 AI research tasks
  from 51 top-tier AI research papers. Evaluations of leading LLM-based agents,
  such as OpenHands and IterativeAgent on EXP-Bench demonstrate partial
  capabilities: while scores on individual experimental aspects such as design or
  implementation correctness occasionally reach 20-35%, the success rate for
  complete, executable experiments was a mere 0.5%. By identifying these
  bottlenecks and providing realistic step-by-step experiment procedures,
  EXP-Bench serves as a vital tool for future AI agents to improve their ability
  to conduct AI research experiments. EXP-Bench is open-sourced at
  https://github.com/Just-Curieous/Curie/tree/main/benchmark/exp_bench.
paper_ai_summary: EXP-Bench evaluates AI agents' end-to-end research experiment capabilities through curated tasks from top AI papers, highlighting current limitations.
paper_ai_keywords: LLM-based agents OpenHands IterativeAgent end-to-end experimentation