SWE-bench Goes Live!
The issue-resolving task, where a model generates patches to fix real-world
bugs, has emerged as a critical benchmark for evaluating the capabilities of
large language models (LLMs). While SWE-bench and its variants have become
standard in this domain, they suffer from key limitations: they have not been
updated since their initial releases, cover a narrow set of repositories, and
depend heavily on manual effort for instance construction and environment
setup. These factors hinder scalability and introduce risks of overfitting and
data contamination. In this work, we present SWE-bench-Live, a
live-updatable benchmark designed to overcome these challenges. Our
initial release consists of 1,319 tasks derived from real GitHub issues created
since 2024, spanning 93 repositories. Each task is accompanied by a dedicated
Docker image to ensure reproducible execution. Central to our benchmark is
\method, an automated curation pipeline that streamlines the entire process
from instance creation to environment setup, removing manual bottlenecks and
enabling scalability and continuous updates. We evaluate a range of
state-of-the-art agent frameworks and LLMs on SWE-bench-Live, revealing a
substantial performance gap compared to static benchmarks like SWE-bench, even
under controlled evaluation conditions. To better understand this discrepancy,
we perform detailed analyses across repository origin, issue recency, and task
difficulty. By providing a fresh, diverse, and executable benchmark grounded in
live repository activity, SWE-bench-Live facilitates rigorous,
contamination-resistant evaluation of LLMs and agents in dynamic, real-world
software development settings.
68391574f85de1fc563cd890
Linghao Zhang
claimed_verified
2025-06-02T07:47:15.958Z
6333e7148a44b83f3f631f81
https://cdn-avatars.huggingface.co/v1/production/uploads/6333e7148a44b83f3f631f81/vFLA69XT0SrdivHUexKzH.jpeg
Linghao Zhang
user
StarryZhang
68391574f85de1fc563cd891
Shilin He
68391574f85de1fc563cd892
Chaoyun Zhang
claimed_verified
2025-06-04T09:03:40.572Z
654dbac9938fbf1e696be8aa
/avatars/b3c4035c48169c1bfb04a439fce3499f.svg
Chaoyun Zhang
user
vyokky
68391574f85de1fc563cd893
Yu Kang
68391574f85de1fc563cd894
Bowen Li
68391574f85de1fc563cd895
Chengxing Xie
68391574f85de1fc563cd896
Junhao Wang
68391574f85de1fc563cd897
Maoquan Wang
68391574f85de1fc563cd898
Yufan Huang
68391574f85de1fc563cd899
Shengyu Fu
68391574f85de1fc563cd89a
Elsie Nallipogu
68391574f85de1fc563cd89b
Qingwei Lin
68391574f85de1fc563cd89c
Yingnong Dang
68391574f85de1fc563cd89d
Saravan Rajmohan
68391574f85de1fc563cd89e
Dongmei Zhang
SWE-bench Goes Live!
The issue-resolving task, where a model generates patches to fix real-world
bugs, has emerged as a critical benchmark for evaluating the capabilities of
large language models (LLMs). While SWE-bench and its variants have become
standard in this domain, they suffer from key limitations: they have not been
updated since their initial releases, cover a narrow set of repositories, and
depend heavily on manual effort for instance construction and environment
setup. These factors hinder scalability and introduce risks of overfitting and
data contamination. In this work, we present SWE-bench-Live, a
live-updatable benchmark designed to overcome these challenges. Our
initial release consists of 1,319 tasks derived from real GitHub issues created
since 2024, spanning 93 repositories. Each task is accompanied by a dedicated
Docker image to ensure reproducible execution. Central to our benchmark is
\method, an automated curation pipeline that streamlines the entire process
from instance creation to environment setup, removing manual bottlenecks and
enabling scalability and continuous updates. We evaluate a range of
state-of-the-art agent frameworks and LLMs on SWE-bench-Live, revealing a
substantial performance gap compared to static benchmarks like SWE-bench, even
under controlled evaluation conditions. To better understand this discrepancy,
we perform detailed analyses across repository origin, issue recency, and task
difficulty. By providing a fresh, diverse, and executable benchmark grounded in
live repository activity, SWE-bench-Live facilitates rigorous,
contamination-resistant evaluation of LLMs and agents in dynamic, real-world
software development settings.
SWE-bench-Live is a continuously updatable benchmark for evaluating LLMs in issue resolution, featuring live GitHub issues and automated curation to ensure scalability and contamination resistance.
large language models LLMs SWE-bench SWE-bench-Live live-updatable benchmark GitHub issues Docker image automated curation pipeline reproducible execution performance evaluation repository origin issue recency task difficulty