Training Software Engineering Agents and Verifiers with SWE-Gym
We present SWE-Gym, the first environment for training real-world software
engineering (SWE) agents. SWE-Gym contains 2,438 real-world Python task
instances, each comprising a codebase with an executable runtime environment,
unit tests, and a task specified in natural language. We use SWE-Gym to train
language model based SWE agents , achieving up to 19% absolute gains in resolve
rate on the popular SWE-Bench Verified and Lite test sets. We also experiment
with inference-time scaling through verifiers trained on agent trajectories
sampled from SWE-Gym. When combined with our fine-tuned SWE agents, we achieve
32.0% and 26.0% on SWE-Bench Verified and Lite, respectively, reflecting a new
state-of-the-art for open-weight SWE agents. To facilitate further research, we
publicly release SWE-Gym, models, and agent trajectories.
67735961702f3c89046839f4
Jiayi Pan
extracted_confirmed
2024-12-31T05:07:53.061Z
61568f37272f2d87a99ba884
https://cdn-avatars.huggingface.co/v1/production/uploads/61568f37272f2d87a99ba884/lgvkl5f0rEyiQRVU5FE32.png
Jiayi Pan
user
Jiayi-Pan
67735961702f3c89046839f5
Xingyao Wang
claimed_verified
2025-01-01T20:13:13.208Z
62c63f31031996c36c848504
https://cdn-avatars.huggingface.co/v1/production/uploads/62c63f31031996c36c848504/xZvH6jnozuFLXbtRSU-h1.jpeg
Xingyao Wang
user
xingyaoww
67735961702f3c89046839f6
Graham Neubig
67735961702f3c89046839f7
Navdeep Jaitly
67735961702f3c89046839f8
Heng Ji
67735961702f3c89046839f9
Alane Suhr
extracted_confirmed
2025-01-02T00:34:15.047Z
6611e6e1188ff298b0dd0b79
/avatars/3a495283955ec9e06e1829c7eb2cd9a4.svg
Alane Suhr
user
alsuhr
67735961702f3c89046839fa
Yizhe Zhang
Training Software Engineering Agents and Verifiers with SWE-Gym
We present SWE-Gym, the first environment for training real-world software
engineering (SWE) agents. SWE-Gym contains 2,438 real-world Python task
instances, each comprising a codebase with an executable runtime environment,
unit tests, and a task specified in natural language. We use SWE-Gym to train
language model based SWE agents , achieving up to 19% absolute gains in resolve
rate on the popular SWE-Bench Verified and Lite test sets. We also experiment
with inference-time scaling through verifiers trained on agent trajectories
sampled from SWE-Gym. When combined with our fine-tuned SWE agents, we achieve
32.0% and 26.0% on SWE-Bench Verified and Lite, respectively, reflecting a new
state-of-the-art for open-weight SWE agents. To facilitate further research, we
publicly release SWE-Gym, models, and agent trajectories.
An environment for training software engineering agents using real-world Python tasks and language models achieves state-of-the-art performance on SWE-Bench.
language model SWE agents SWE-Gym SWE-Bench Verified SWE-Bench Lite agent trajectories inference-time scaling verifiers