title: SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research
    Repositories
summary: Given that Large Language Models (LLMs) have made significant progress in
  writing code, can they now be used to autonomously reproduce results from
  research repositories? Such a capability would be a boon to the research
  community, helping researchers validate, understand, and extend prior work. To
  advance towards this goal, we introduce SUPER, the first benchmark designed to
  evaluate the capability of LLMs in setting up and executing tasks from research
  repositories. SUPERaims to capture the realistic challenges faced by
  researchers working with Machine Learning (ML) and Natural Language Processing
  (NLP) research repositories. Our benchmark comprises three distinct problem
  sets: 45 end-to-end problems with annotated expert solutions, 152 sub problems
  derived from the expert set that focus on specific challenges (e.g.,
  configuring a trainer), and 602 automatically generated problems for
  larger-scale development. We introduce various evaluation measures to assess
  both task success and progress, utilizing gold solutions when available or
  approximations otherwise. We show that state-of-the-art approaches struggle to
  solve these problems with the best model (GPT-4o) solving only 16.3% of the
  end-to-end set, and 46.1% of the scenarios. This illustrates the challenge of
  this task, and suggests that SUPER can serve as a valuable resource for the
  community to make and measure progress.
paper_authors._id: 66e33a5563a8b976f289dc16
paper_authors.name: Ben Bogin
paper_authors._id: 66e33a5563a8b976f289dc17
paper_authors.name: Kejuan Yang
paper_authors._id: 66e33a5563a8b976f289dc18
paper_authors.name: Shashank Gupta
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-13T07:20:56.080Z
paper_authors._id: 5e9d15b74957053f60648a34
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5e9d15b74957053f60648a34/_Nt3HWtz6moVQIfa5ieIJ.jpeg
paper_authors.fullname: Shashank Gupta
paper_authors.type: user
paper_authors.user: shatu
paper_authors._id: 66e33a5563a8b976f289dc19
paper_authors.name: Kyle Richardson
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-03T14:24:13.885Z
paper_authors._id: 659c6f5e1d398a238152227d
paper_authors.avatarUrl: /avatars/dba97fe8cb825102f1eae97104a71f64.svg
paper_authors.fullname: Kyle Richardson
paper_authors.type: user
paper_authors.user: yakazimir
paper_authors._id: 66e33a5563a8b976f289dc1a
paper_authors.name: Erin Bransom
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-26T21:10:49.876Z
paper_authors._id: 65bd91cfb54ab5b37dd1e309
paper_authors.avatarUrl: /avatars/33a26756df8275b94e8e4f1f8a6e8cbc.svg
paper_authors.fullname: Erin Bransom
paper_authors.type: user
paper_authors.user: erinbransom
paper_authors._id: 66e33a5563a8b976f289dc1b
paper_authors.name: Peter Clark
paper_authors._id: 66e33a5563a8b976f289dc1c
paper_authors.name: Ashish Sabharwal
paper_authors._id: 66e33a5563a8b976f289dc1d
paper_authors.name: Tushar Khot
paper_title: SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research
    Repositories
paper_summary: Given that Large Language Models (LLMs) have made significant progress in
  writing code, can they now be used to autonomously reproduce results from
  research repositories? Such a capability would be a boon to the research
  community, helping researchers validate, understand, and extend prior work. To
  advance towards this goal, we introduce SUPER, the first benchmark designed to
  evaluate the capability of LLMs in setting up and executing tasks from research
  repositories. SUPERaims to capture the realistic challenges faced by
  researchers working with Machine Learning (ML) and Natural Language Processing
  (NLP) research repositories. Our benchmark comprises three distinct problem
  sets: 45 end-to-end problems with annotated expert solutions, 152 sub problems
  derived from the expert set that focus on specific challenges (e.g.,
  configuring a trainer), and 602 automatically generated problems for
  larger-scale development. We introduce various evaluation measures to assess
  both task success and progress, utilizing gold solutions when available or
  approximations otherwise. We show that state-of-the-art approaches struggle to
  solve these problems with the best model (GPT-4o) solving only 16.3% of the
  end-to-end set, and 46.1% of the scenarios. This illustrates the challenge of
  this task, and suggests that SUPER can serve as a valuable resource for the
  community to make and measure progress.
paper_ai_summary: SUPER, a novel benchmark, evaluates Large Language Models' ability to autonomously set up and execute tasks from research repositories, highlighting significant challenges in this domain.
paper_ai_keywords: Large Language Models SUPER benchmark end-to-end problems sub problems automatically generated problems GPT-4o task success progress gold solutions approximations Machine Learning Natural Language Processing