title: SWE-bench Goes Live!
summary: The issue-resolving task, where a model generates patches to fix real-world
  bugs, has emerged as a critical benchmark for evaluating the capabilities of
  large language models (LLMs). While SWE-bench and its variants have become
  standard in this domain, they suffer from key limitations: they have not been
  updated since their initial releases, cover a narrow set of repositories, and
  depend heavily on manual effort for instance construction and environment
  setup. These factors hinder scalability and introduce risks of overfitting and
  data contamination. In this work, we present SWE-bench-Live, a
  live-updatable benchmark designed to overcome these challenges. Our
  initial release consists of 1,319 tasks derived from real GitHub issues created
  since 2024, spanning 93 repositories. Each task is accompanied by a dedicated
  Docker image to ensure reproducible execution. Central to our benchmark is
  \method, an automated curation pipeline that streamlines the entire process
  from instance creation to environment setup, removing manual bottlenecks and
  enabling scalability and continuous updates. We evaluate a range of
  state-of-the-art agent frameworks and LLMs on SWE-bench-Live, revealing a
  substantial performance gap compared to static benchmarks like SWE-bench, even
  under controlled evaluation conditions. To better understand this discrepancy,
  we perform detailed analyses across repository origin, issue recency, and task
  difficulty. By providing a fresh, diverse, and executable benchmark grounded in
  live repository activity, SWE-bench-Live facilitates rigorous,
  contamination-resistant evaluation of LLMs and agents in dynamic, real-world
  software development settings.
paper_authors._id: 68391574f85de1fc563cd890
paper_authors.name: Linghao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:47:15.958Z
paper_authors._id: 6333e7148a44b83f3f631f81
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6333e7148a44b83f3f631f81/vFLA69XT0SrdivHUexKzH.jpeg
paper_authors.fullname: Linghao Zhang
paper_authors.type: user
paper_authors.user: StarryZhang
paper_authors._id: 68391574f85de1fc563cd891
paper_authors.name: Shilin He
paper_authors._id: 68391574f85de1fc563cd892
paper_authors.name: Chaoyun Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T09:03:40.572Z
paper_authors._id: 654dbac9938fbf1e696be8aa
paper_authors.avatarUrl: /avatars/b3c4035c48169c1bfb04a439fce3499f.svg
paper_authors.fullname: Chaoyun Zhang
paper_authors.type: user
paper_authors.user: vyokky
paper_authors._id: 68391574f85de1fc563cd893
paper_authors.name: Yu Kang
paper_authors._id: 68391574f85de1fc563cd894
paper_authors.name: Bowen Li
paper_authors._id: 68391574f85de1fc563cd895
paper_authors.name: Chengxing Xie
paper_authors._id: 68391574f85de1fc563cd896
paper_authors.name: Junhao Wang
paper_authors._id: 68391574f85de1fc563cd897
paper_authors.name: Maoquan Wang
paper_authors._id: 68391574f85de1fc563cd898
paper_authors.name: Yufan Huang
paper_authors._id: 68391574f85de1fc563cd899
paper_authors.name: Shengyu Fu
paper_authors._id: 68391574f85de1fc563cd89a
paper_authors.name: Elsie Nallipogu
paper_authors._id: 68391574f85de1fc563cd89b
paper_authors.name: Qingwei Lin
paper_authors._id: 68391574f85de1fc563cd89c
paper_authors.name: Yingnong Dang
paper_authors._id: 68391574f85de1fc563cd89d
paper_authors.name: Saravan Rajmohan
paper_authors._id: 68391574f85de1fc563cd89e
paper_authors.name: Dongmei Zhang
paper_title: SWE-bench Goes Live!
paper_summary: The issue-resolving task, where a model generates patches to fix real-world
  bugs, has emerged as a critical benchmark for evaluating the capabilities of
  large language models (LLMs). While SWE-bench and its variants have become
  standard in this domain, they suffer from key limitations: they have not been
  updated since their initial releases, cover a narrow set of repositories, and
  depend heavily on manual effort for instance construction and environment
  setup. These factors hinder scalability and introduce risks of overfitting and
  data contamination. In this work, we present SWE-bench-Live, a
  live-updatable benchmark designed to overcome these challenges. Our
  initial release consists of 1,319 tasks derived from real GitHub issues created
  since 2024, spanning 93 repositories. Each task is accompanied by a dedicated
  Docker image to ensure reproducible execution. Central to our benchmark is
  \method, an automated curation pipeline that streamlines the entire process
  from instance creation to environment setup, removing manual bottlenecks and
  enabling scalability and continuous updates. We evaluate a range of
  state-of-the-art agent frameworks and LLMs on SWE-bench-Live, revealing a
  substantial performance gap compared to static benchmarks like SWE-bench, even
  under controlled evaluation conditions. To better understand this discrepancy,
  we perform detailed analyses across repository origin, issue recency, and task
  difficulty. By providing a fresh, diverse, and executable benchmark grounded in
  live repository activity, SWE-bench-Live facilitates rigorous,
  contamination-resistant evaluation of LLMs and agents in dynamic, real-world
  software development settings.
paper_ai_summary: SWE-bench-Live is a continuously updatable benchmark for evaluating LLMs in issue resolution, featuring live GitHub issues and automated curation to ensure scalability and contamination resistance.
paper_ai_keywords: large language models LLMs SWE-bench SWE-bench-Live live-updatable benchmark GitHub issues Docker image automated curation pipeline reproducible execution performance evaluation repository origin issue recency task difficulty