CodeMonkeys: Scaling Test-Time Compute for Software Engineering
Scaling test-time compute is a promising axis for improving LLM capabilities.
However, test-time compute can be scaled in a variety of ways, and effectively
combining different approaches remains an active area of research. Here, we
explore this problem in the context of solving real-world GitHub issues from
the SWE-bench dataset. Our system, named CodeMonkeys, allows models to
iteratively edit a codebase by jointly generating and running a testing script
alongside their draft edit. We sample many of these multi-turn trajectories for
every issue to generate a collection of candidate edits. This approach lets us
scale "serial" test-time compute by increasing the number of iterations per
trajectory and "parallel" test-time compute by increasing the number of
trajectories per problem. With parallel scaling, we can amortize up-front costs
across multiple downstream samples, allowing us to identify relevant codebase
context using the simple method of letting an LLM read every file. In order to
select between candidate edits, we combine voting using model-generated tests
with a final multi-turn trajectory dedicated to selection. Overall, CodeMonkeys
resolves 57.4% of issues from SWE-bench Verified using a budget of
approximately 2300 USD. Our selection method can also be used to combine
candidates from different sources. Selecting over an ensemble of edits from
existing top SWE-bench Verified submissions obtains a score of 66.2% and
outperforms the best member of the ensemble on its own. We fully release our
code and data at https://scalingintelligence.stanford.edu/pubs/codemonkeys.
67991502dc9404d4424ce38c
Ryan Ehrlich
67991502dc9404d4424ce38d
Bradley Brown
67991502dc9404d4424ce38e
Jordan Juravsky
67991502dc9404d4424ce38f
Ronald Clark
67991502dc9404d4424ce390
Christopher Ré
67991502dc9404d4424ce391
Azalia Mirhoseini
extracted_confirmed
2025-02-02T01:37:18.386Z
66ba4e72c9b2ab14b3707be0
/avatars/97ef14d683ed2d0115a9c4694b9763dc.svg
Azalia Mirhoseini
user
am34
CodeMonkeys: Scaling Test-Time Compute for Software Engineering
Scaling test-time compute is a promising axis for improving LLM capabilities.
However, test-time compute can be scaled in a variety of ways, and effectively
combining different approaches remains an active area of research. Here, we
explore this problem in the context of solving real-world GitHub issues from
the SWE-bench dataset. Our system, named CodeMonkeys, allows models to
iteratively edit a codebase by jointly generating and running a testing script
alongside their draft edit. We sample many of these multi-turn trajectories for
every issue to generate a collection of candidate edits. This approach lets us
scale "serial" test-time compute by increasing the number of iterations per
trajectory and "parallel" test-time compute by increasing the number of
trajectories per problem. With parallel scaling, we can amortize up-front costs
across multiple downstream samples, allowing us to identify relevant codebase
context using the simple method of letting an LLM read every file. In order to
select between candidate edits, we combine voting using model-generated tests
with a final multi-turn trajectory dedicated to selection. Overall, CodeMonkeys
resolves 57.4% of issues from SWE-bench Verified using a budget of
approximately 2300 USD. Our selection method can also be used to combine
candidates from different sources. Selecting over an ensemble of edits from
existing top SWE-bench Verified submissions obtains a score of 66.2% and
outperforms the best member of the ensemble on its own. We fully release our
code and data at https://scalingintelligence.stanford.edu/pubs/codemonkeys.
CodeMonkeys iteratively generates and tests code edits to resolve real-world GitHub issues, scaling test-time compute through multiple iterations and parallel trajectories, and outperforms individual models by combining candidate solutions.
test-time compute LLM capabilities iterative editing test-time compute scaling multi-turn trajectories candidate edits parallel scaling up-front costs model-generated tests SWE-bench CodeMonkeys