SWE-rebench: An Automated Pipeline for Task Collection and
  Decontaminated Evaluation of Software Engineering Agents
LLM-based agents have shown promising capabilities in a growing range of
software engineering (SWE) tasks. However, advancing this field faces two
critical challenges. First, high-quality training data is scarce, especially
data that reflects real-world SWE scenarios, where agents must interact with
development environments, execute code and adapt behavior based on the outcomes
of their actions. Existing datasets are either limited to one-shot code
generation or comprise small, manually curated collections of interactive
tasks, lacking both scale and diversity. Second, the lack of fresh interactive
SWE tasks affects evaluation of rapidly improving models, as static benchmarks
quickly become outdated due to contamination issues. To address these
limitations, we introduce a novel, automated, and scalable pipeline to
continuously extract real-world interactive SWE tasks from diverse GitHub
repositories. Using this pipeline, we construct SWE-rebench, a public dataset
comprising over 21,000 interactive Python-based SWE tasks, suitable for
reinforcement learning of SWE agents at scale. Additionally, we use continuous
supply of fresh tasks collected using SWE-rebench methodology to build a
contamination-free benchmark for agentic software engineering. We compare
results of various LLMs on this benchmark to results on SWE-bench Verified and
show that performance of some language models might be inflated due to
contamination issues.
683735aff42cc8a1d260e677
Ibragim Badertdinov
claimed_verified
2025-05-29T09:40:24.705Z
654e5e094319c75e3e1b6cbc
/avatars/a8889036fa38f80f2d45aea8d1471395.svg
Ibragim
user
ibragim-bad
683735aff42cc8a1d260e678
Alexander Golubev
claimed_verified
2025-05-28T19:13:38.030Z
644e9ffcd6001776ed77d874
/avatars/b93e02caf929679b7e9bc589eed0b689.svg
Alexander
user
djalexj
683735aff42cc8a1d260e679
Maksim Nekrashevich
683735aff42cc8a1d260e67a
Anton Shevtsov
claimed_verified
2025-05-29T20:18:57.150Z
66214f06d55e5f677e0da9f2
/avatars/dc3b8de4a72b0d10f6bb90594488a9ec.svg
Anton
user
mrshevan
683735aff42cc8a1d260e67b
Simon Karasik
claimed_verified
2025-05-29T09:40:26.644Z
65e48cb3a4e46e644ec1277d
/avatars/dd2bf04a6f81bf0a0892080af5d485b2.svg
Simon Karasik
user
sbkarasik
683735aff42cc8a1d260e67c
Andrei Andriushchenko
claimed_verified
2025-06-02T07:48:37.439Z
65dc8a36d4cb1534acf98553
/avatars/0f39967027ce8e04de4584c6bfc0ddf6.svg
Andrei Andriushchenko
user
devkettle
683735aff42cc8a1d260e67d
Maria Trofimova
683735aff42cc8a1d260e67e
Daria Litvintseva
683735aff42cc8a1d260e67f
Boris Yangel
claimed_verified
2025-06-02T07:48:34.554Z
63d83e6c6ac3104e50c8dcd6
/avatars/2a4b82aa3fcd0617387c6504da17b071.svg
Boris Yangel
user
hr0nix
SWE-rebench: An Automated Pipeline for Task Collection and
  Decontaminated Evaluation of Software Engineering Agents
LLM-based agents have shown promising capabilities in a growing range of
software engineering (SWE) tasks. However, advancing this field faces two
critical challenges. First, high-quality training data is scarce, especially
data that reflects real-world SWE scenarios, where agents must interact with
development environments, execute code and adapt behavior based on the outcomes
of their actions. Existing datasets are either limited to one-shot code
generation or comprise small, manually curated collections of interactive
tasks, lacking both scale and diversity. Second, the lack of fresh interactive
SWE tasks affects evaluation of rapidly improving models, as static benchmarks
quickly become outdated due to contamination issues. To address these
limitations, we introduce a novel, automated, and scalable pipeline to
continuously extract real-world interactive SWE tasks from diverse GitHub
repositories. Using this pipeline, we construct SWE-rebench, a public dataset
comprising over 21,000 interactive Python-based SWE tasks, suitable for
reinforcement learning of SWE agents at scale. Additionally, we use continuous
supply of fresh tasks collected using SWE-rebench methodology to build a
contamination-free benchmark for agentic software engineering. We compare
results of various LLMs on this benchmark to results on SWE-bench Verified and
show that performance of some language models might be inflated due to
contamination issues.
A novel pipeline extracts real-world, interactive software engineering tasks from GitHub to create SWE-rebench, improving the evaluation of reinforcement learning models in SWE.
LLM-based agents reinforcement learning software engineering tasks GitHub repositories SWE-rebench contamination-free benchmark SWE-bench Verified