SWE-Factory: Your Automated Factory for Issue Resolution Training Data
  and Evaluation Benchmarks
Constructing large-scale datasets for the GitHub issue resolution task is
crucial for both training and evaluating the software engineering capabilities
of Large Language Models (LLMs). However, the traditional process for creating
such benchmarks is notoriously challenging and labor-intensive, particularly in
the stages of setting up evaluation environments, grading test outcomes, and
validating task instances. In this paper, we propose SWE-Factory, an automated
pipeline designed to address these challenges. To tackle these issues, our
pipeline integrates three core automated components. First, we introduce
SWE-Builder, a multi-agent system that automates evaluation environment
construction, which employs four specialized agents that work in a
collaborative, iterative loop and leverages an environment memory pool to
enhance efficiency. Second, we introduce a standardized, exit-code-based
grading method that eliminates the need for manually writing custom parsers.
Finally, we automate the fail2pass validation process using these reliable exit
code signals. Experiments on 671 issues across four programming languages show
that our pipeline can effectively construct valid task instances; for example,
with GPT-4.1-mini, our SWE-Builder constructs 269 valid instances at 0.045 per
instance, while with Gemini-2.5-flash, it achieves comparable performance at
the lowest cost of 0.024 per instance. We also demonstrate that our
exit-code-based grading achieves 100% accuracy compared to manual inspection,
and our automated fail2pass validation reaches a precision of 0.92 and a recall
of 1.00. We hope our automated pipeline will accelerate the collection of
large-scale, high-quality GitHub issue resolution datasets for both training
and evaluation. Our code and datasets are released at
https://github.com/DeepSoftwareAnalytics/swe-factory.
684b7ea83b733ba333686f8a
Lianghong Guo
684b7ea83b733ba333686f8b
Yanlin Wang
684b7ea83b733ba333686f8c
Caihua Li
684b7ea83b733ba333686f8d
Pengyu Yang
684b7ea83b733ba333686f8e
Jiachi Chen
684b7ea83b733ba333686f8f
Wei Tao
claimed_verified
2025-06-13T07:39:58.091Z
6355473d525beaee688b7ba1
/avatars/1fb0d57ed5f1a9b872a1ada8b2973ffb.svg
Wei Tao
user
itaowe
684b7ea83b733ba333686f90
Yingtian Zou
684b7ea83b733ba333686f91
Duyu Tang
684b7ea83b733ba333686f92
Zibin Zheng
SWE-Factory: Your Automated Factory for Issue Resolution Training Data
  and Evaluation Benchmarks
Constructing large-scale datasets for the GitHub issue resolution task is
crucial for both training and evaluating the software engineering capabilities
of Large Language Models (LLMs). However, the traditional process for creating
such benchmarks is notoriously challenging and labor-intensive, particularly in
the stages of setting up evaluation environments, grading test outcomes, and
validating task instances. In this paper, we propose SWE-Factory, an automated
pipeline designed to address these challenges. To tackle these issues, our
pipeline integrates three core automated components. First, we introduce
SWE-Builder, a multi-agent system that automates evaluation environment
construction, which employs four specialized agents that work in a
collaborative, iterative loop and leverages an environment memory pool to
enhance efficiency. Second, we introduce a standardized, exit-code-based
grading method that eliminates the need for manually writing custom parsers.
Finally, we automate the fail2pass validation process using these reliable exit
code signals. Experiments on 671 issues across four programming languages show
that our pipeline can effectively construct valid task instances; for example,
with GPT-4.1-mini, our SWE-Builder constructs 269 valid instances at 0.045 per
instance, while with Gemini-2.5-flash, it achieves comparable performance at
the lowest cost of 0.024 per instance. We also demonstrate that our
exit-code-based grading achieves 100% accuracy compared to manual inspection,
and our automated fail2pass validation reaches a precision of 0.92 and a recall
of 1.00. We hope our automated pipeline will accelerate the collection of
large-scale, high-quality GitHub issue resolution datasets for both training
and evaluation. Our code and datasets are released at
https://github.com/DeepSoftwareAnalytics/swe-factory.
A pipeline named SWE-Factory automates the creation and validation of GitHub issue resolution datasets for training and evaluating Large Language Models, using SWE-Builder for environment setup, exit-code-based grading, and automated fail2pass validation.
Large Language Models LLMs SWE-Factory SWE-Builder multi-agent system environment memory pool exit-code-based grading automated fail2pass validation GPT-4.1-mini Gemini-2.5-flash precision recall