title: CodeMonkeys: Scaling Test-Time Compute for Software Engineering
summary: Scaling test-time compute is a promising axis for improving LLM capabilities.
  However, test-time compute can be scaled in a variety of ways, and effectively
  combining different approaches remains an active area of research. Here, we
  explore this problem in the context of solving real-world GitHub issues from
  the SWE-bench dataset. Our system, named CodeMonkeys, allows models to
  iteratively edit a codebase by jointly generating and running a testing script
  alongside their draft edit. We sample many of these multi-turn trajectories for
  every issue to generate a collection of candidate edits. This approach lets us
  scale "serial" test-time compute by increasing the number of iterations per
  trajectory and "parallel" test-time compute by increasing the number of
  trajectories per problem. With parallel scaling, we can amortize up-front costs
  across multiple downstream samples, allowing us to identify relevant codebase
  context using the simple method of letting an LLM read every file. In order to
  select between candidate edits, we combine voting using model-generated tests
  with a final multi-turn trajectory dedicated to selection. Overall, CodeMonkeys
  resolves 57.4% of issues from SWE-bench Verified using a budget of
  approximately 2300 USD. Our selection method can also be used to combine
  candidates from different sources. Selecting over an ensemble of edits from
  existing top SWE-bench Verified submissions obtains a score of 66.2% and
  outperforms the best member of the ensemble on its own. We fully release our
  code and data at https://scalingintelligence.stanford.edu/pubs/codemonkeys.
paper_authors._id: 67991502dc9404d4424ce38c
paper_authors.name: Ryan Ehrlich
paper_authors._id: 67991502dc9404d4424ce38d
paper_authors.name: Bradley Brown
paper_authors._id: 67991502dc9404d4424ce38e
paper_authors.name: Jordan Juravsky
paper_authors._id: 67991502dc9404d4424ce38f
paper_authors.name: Ronald Clark
paper_authors._id: 67991502dc9404d4424ce390
paper_authors.name: Christopher Ré
paper_authors._id: 67991502dc9404d4424ce391
paper_authors.name: Azalia Mirhoseini
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-02-02T01:37:18.386Z
paper_authors._id: 66ba4e72c9b2ab14b3707be0
paper_authors.avatarUrl: /avatars/97ef14d683ed2d0115a9c4694b9763dc.svg
paper_authors.fullname: Azalia Mirhoseini
paper_authors.type: user
paper_authors.user: am34
paper_title: CodeMonkeys: Scaling Test-Time Compute for Software Engineering
paper_summary: Scaling test-time compute is a promising axis for improving LLM capabilities.
  However, test-time compute can be scaled in a variety of ways, and effectively
  combining different approaches remains an active area of research. Here, we
  explore this problem in the context of solving real-world GitHub issues from
  the SWE-bench dataset. Our system, named CodeMonkeys, allows models to
  iteratively edit a codebase by jointly generating and running a testing script
  alongside their draft edit. We sample many of these multi-turn trajectories for
  every issue to generate a collection of candidate edits. This approach lets us
  scale "serial" test-time compute by increasing the number of iterations per
  trajectory and "parallel" test-time compute by increasing the number of
  trajectories per problem. With parallel scaling, we can amortize up-front costs
  across multiple downstream samples, allowing us to identify relevant codebase
  context using the simple method of letting an LLM read every file. In order to
  select between candidate edits, we combine voting using model-generated tests
  with a final multi-turn trajectory dedicated to selection. Overall, CodeMonkeys
  resolves 57.4% of issues from SWE-bench Verified using a budget of
  approximately 2300 USD. Our selection method can also be used to combine
  candidates from different sources. Selecting over an ensemble of edits from
  existing top SWE-bench Verified submissions obtains a score of 66.2% and
  outperforms the best member of the ensemble on its own. We fully release our
  code and data at https://scalingintelligence.stanford.edu/pubs/codemonkeys.
paper_ai_keywords: LLM (Large Language Model) modalities SWE-bench (Software Engineering Benchmark) GitHub issues iterative editing testing script multi-turn trajectories test-time compute parallel scaling up-front costs codebase context model-generated tests SWE-bench Verified ensemble of edits