SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research
  Repositories
Given that Large Language Models (LLMs) have made significant progress in
writing code, can they now be used to autonomously reproduce results from
research repositories? Such a capability would be a boon to the research
community, helping researchers validate, understand, and extend prior work. To
advance towards this goal, we introduce SUPER, the first benchmark designed to
evaluate the capability of LLMs in setting up and executing tasks from research
repositories. SUPERaims to capture the realistic challenges faced by
researchers working with Machine Learning (ML) and Natural Language Processing
(NLP) research repositories. Our benchmark comprises three distinct problem
sets: 45 end-to-end problems with annotated expert solutions, 152 sub problems
derived from the expert set that focus on specific challenges (e.g.,
configuring a trainer), and 602 automatically generated problems for
larger-scale development. We introduce various evaluation measures to assess
both task success and progress, utilizing gold solutions when available or
approximations otherwise. We show that state-of-the-art approaches struggle to
solve these problems with the best model (GPT-4o) solving only 16.3% of the
end-to-end set, and 46.1% of the scenarios. This illustrates the challenge of
this task, and suggests that SUPER can serve as a valuable resource for the
community to make and measure progress.
66e33a5563a8b976f289dc16
Ben Bogin
66e33a5563a8b976f289dc17
Kejuan Yang
66e33a5563a8b976f289dc18
Shashank Gupta
claimed_verified
2024-09-13T07:20:56.080Z
5e9d15b74957053f60648a34
https://cdn-avatars.huggingface.co/v1/production/uploads/5e9d15b74957053f60648a34/_Nt3HWtz6moVQIfa5ieIJ.jpeg
Shashank Gupta
user
shatu
66e33a5563a8b976f289dc19
Kyle Richardson
claimed_verified
2024-11-03T14:24:13.885Z
659c6f5e1d398a238152227d
/avatars/dba97fe8cb825102f1eae97104a71f64.svg
Kyle Richardson
user
yakazimir
66e33a5563a8b976f289dc1a
Erin Bransom
claimed_verified
2024-09-26T21:10:49.876Z
65bd91cfb54ab5b37dd1e309
/avatars/33a26756df8275b94e8e4f1f8a6e8cbc.svg
Erin Bransom
user
erinbransom
66e33a5563a8b976f289dc1b
Peter Clark
66e33a5563a8b976f289dc1c
Ashish Sabharwal
claimed_verified
2025-07-22T07:57:35.091Z
687e87f9b193d0f4c8d605c0
/avatars/1e2fa294cbb2017e380fae6695e35db1.svg
Ashish Sabharwal
user
ashish333
66e33a5563a8b976f289dc1d
Tushar Khot
SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research
  Repositories
Given that Large Language Models (LLMs) have made significant progress in
writing code, can they now be used to autonomously reproduce results from
research repositories? Such a capability would be a boon to the research
community, helping researchers validate, understand, and extend prior work. To
advance towards this goal, we introduce SUPER, the first benchmark designed to
evaluate the capability of LLMs in setting up and executing tasks from research
repositories. SUPERaims to capture the realistic challenges faced by
researchers working with Machine Learning (ML) and Natural Language Processing
(NLP) research repositories. Our benchmark comprises three distinct problem
sets: 45 end-to-end problems with annotated expert solutions, 152 sub problems
derived from the expert set that focus on specific challenges (e.g.,
configuring a trainer), and 602 automatically generated problems for
larger-scale development. We introduce various evaluation measures to assess
both task success and progress, utilizing gold solutions when available or
approximations otherwise. We show that state-of-the-art approaches struggle to
solve these problems with the best model (GPT-4o) solving only 16.3% of the
end-to-end set, and 46.1% of the scenarios. This illustrates the challenge of
this task, and suggests that SUPER can serve as a valuable resource for the
community to make and measure progress.
SUPER, a novel benchmark, evaluates Large Language Models' ability to autonomously set up and execute tasks from research repositories, highlighting significant challenges in this domain.
Large Language Models SUPER benchmark end-to-end problems sub problems automatically generated problems GPT-4o task success progress gold solutions approximations Machine Learning Natural Language Processing