title: SWE-rebench: An Automated Pipeline for Task Collection and
    Decontaminated Evaluation of Software Engineering Agents
summary: LLM-based agents have shown promising capabilities in a growing range of
  software engineering (SWE) tasks. However, advancing this field faces two
  critical challenges. First, high-quality training data is scarce, especially
  data that reflects real-world SWE scenarios, where agents must interact with
  development environments, execute code and adapt behavior based on the outcomes
  of their actions. Existing datasets are either limited to one-shot code
  generation or comprise small, manually curated collections of interactive
  tasks, lacking both scale and diversity. Second, the lack of fresh interactive
  SWE tasks affects evaluation of rapidly improving models, as static benchmarks
  quickly become outdated due to contamination issues. To address these
  limitations, we introduce a novel, automated, and scalable pipeline to
  continuously extract real-world interactive SWE tasks from diverse GitHub
  repositories. Using this pipeline, we construct SWE-rebench, a public dataset
  comprising over 21,000 interactive Python-based SWE tasks, suitable for
  reinforcement learning of SWE agents at scale. Additionally, we use continuous
  supply of fresh tasks collected using SWE-rebench methodology to build a
  contamination-free benchmark for agentic software engineering. We compare
  results of various LLMs on this benchmark to results on SWE-bench Verified and
  show that performance of some language models might be inflated due to
  contamination issues.
paper_authors._id: 683735aff42cc8a1d260e677
paper_authors.name: Ibragim Badertdinov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T09:40:24.705Z
paper_authors._id: 654e5e094319c75e3e1b6cbc
paper_authors.avatarUrl: /avatars/a8889036fa38f80f2d45aea8d1471395.svg
paper_authors.fullname: Ibragim
paper_authors.type: user
paper_authors.user: ibragim-bad
paper_authors._id: 683735aff42cc8a1d260e678
paper_authors.name: Alexander Golubev
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T19:13:38.030Z
paper_authors._id: 644e9ffcd6001776ed77d874
paper_authors.avatarUrl: /avatars/b93e02caf929679b7e9bc589eed0b689.svg
paper_authors.fullname: Alexander
paper_authors.type: user
paper_authors.user: djalexj
paper_authors._id: 683735aff42cc8a1d260e679
paper_authors.name: Maksim Nekrashevich
paper_authors._id: 683735aff42cc8a1d260e67a
paper_authors.name: Anton Shevtsov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T20:18:57.150Z
paper_authors._id: 66214f06d55e5f677e0da9f2
paper_authors.avatarUrl: /avatars/dc3b8de4a72b0d10f6bb90594488a9ec.svg
paper_authors.fullname: Anton
paper_authors.type: user
paper_authors.user: mrshevan
paper_authors._id: 683735aff42cc8a1d260e67b
paper_authors.name: Simon Karasik
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T09:40:26.644Z
paper_authors._id: 65e48cb3a4e46e644ec1277d
paper_authors.avatarUrl: /avatars/dd2bf04a6f81bf0a0892080af5d485b2.svg
paper_authors.fullname: Simon Karasik
paper_authors.type: user
paper_authors.user: sbkarasik
paper_authors._id: 683735aff42cc8a1d260e67c
paper_authors.name: Andrei Andriushchenko
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:48:37.439Z
paper_authors._id: 65dc8a36d4cb1534acf98553
paper_authors.avatarUrl: /avatars/0f39967027ce8e04de4584c6bfc0ddf6.svg
paper_authors.fullname: Andrei Andriushchenko
paper_authors.type: user
paper_authors.user: devkettle
paper_authors._id: 683735aff42cc8a1d260e67d
paper_authors.name: Maria Trofimova
paper_authors._id: 683735aff42cc8a1d260e67e
paper_authors.name: Daria Litvintseva
paper_authors._id: 683735aff42cc8a1d260e67f
paper_authors.name: Boris Yangel
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:48:34.554Z
paper_authors._id: 63d83e6c6ac3104e50c8dcd6
paper_authors.avatarUrl: /avatars/2a4b82aa3fcd0617387c6504da17b071.svg
paper_authors.fullname: Boris Yangel
paper_authors.type: user
paper_authors.user: hr0nix
paper_title: SWE-rebench: An Automated Pipeline for Task Collection and
    Decontaminated Evaluation of Software Engineering Agents
paper_summary: LLM-based agents have shown promising capabilities in a growing range of
  software engineering (SWE) tasks. However, advancing this field faces two
  critical challenges. First, high-quality training data is scarce, especially
  data that reflects real-world SWE scenarios, where agents must interact with
  development environments, execute code and adapt behavior based on the outcomes
  of their actions. Existing datasets are either limited to one-shot code
  generation or comprise small, manually curated collections of interactive
  tasks, lacking both scale and diversity. Second, the lack of fresh interactive
  SWE tasks affects evaluation of rapidly improving models, as static benchmarks
  quickly become outdated due to contamination issues. To address these
  limitations, we introduce a novel, automated, and scalable pipeline to
  continuously extract real-world interactive SWE tasks from diverse GitHub
  repositories. Using this pipeline, we construct SWE-rebench, a public dataset
  comprising over 21,000 interactive Python-based SWE tasks, suitable for
  reinforcement learning of SWE agents at scale. Additionally, we use continuous
  supply of fresh tasks collected using SWE-rebench methodology to build a
  contamination-free benchmark for agentic software engineering. We compare
  results of various LLMs on this benchmark to results on SWE-bench Verified and
  show that performance of some language models might be inflated due to
  contamination issues.
paper_ai_summary: A novel pipeline extracts real-world, interactive software engineering tasks from GitHub to create SWE-rebench, improving the evaluation of reinforcement learning models in SWE.
paper_ai_keywords: LLM-based agents reinforcement learning software engineering tasks GitHub repositories SWE-rebench contamination-free benchmark SWE-bench Verified