title: Satori-SWE: Evolutionary Test-Time Scaling for Sample-Efficient Software
    Engineering
summary: Language models (LMs) perform well on standardized coding benchmarks but
  struggle with real-world software engineering tasks such as resolving GitHub
  issues in SWE-Bench, especially when model parameters are less than 100B. While
  smaller models are preferable in practice due to their lower computational
  cost, improving their performance remains challenging. Existing approaches
  primarily rely on supervised fine-tuning (SFT) with high-quality data, which is
  expensive to curate at scale. An alternative is test-time scaling: generating
  multiple outputs, scoring them using a verifier, and selecting the best one.
  Although effective, this strategy often requires excessive sampling and costly
  scoring, limiting its practical application. We propose Evolutionary Test-Time
  Scaling (EvoScale), a sample-efficient method that treats generation as an
  evolutionary process. By iteratively refining outputs via selection and
  mutation, EvoScale shifts the output distribution toward higher-scoring
  regions, reducing the number of samples needed to find correct solutions. To
  reduce the overhead from repeatedly sampling and selection, we train the model
  to self-evolve using reinforcement learning (RL). Rather than relying on
  external verifiers at inference time, the model learns to self-improve the
  scores of its own generations across iterations. Evaluated on
  SWE-Bench-Verified, EvoScale enables our 32B model, Satori-SWE-32B, to match or
  exceed the performance of models with over 100B parameters while using a few
  samples. Code, data, and models will be fully open-sourced.
paper_authors._id: 68390da8f527444e97c4ab95
paper_authors.name: Guangtao Zeng
paper_authors._id: 68390da8f527444e97c4ab96
paper_authors.name: Maohao Shen
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-05-30T01:45:13.275Z
paper_authors._id: 6553c985a7aded0380b5f928
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6553c985a7aded0380b5f928/tAJnmYvmaAse4fbHq0CjA.jpeg
paper_authors.fullname: Maohao Shen
paper_authors.type: user
paper_authors.user: maohaos2
paper_authors._id: 68390da8f527444e97c4ab97
paper_authors.name: Delin Chen
paper_authors._id: 68390da8f527444e97c4ab98
paper_authors.name: Zhenting Qi
paper_authors._id: 68390da8f527444e97c4ab99
paper_authors.name: Subhro Das
paper_authors._id: 68390da8f527444e97c4ab9a
paper_authors.name: Dan Gutfreund
paper_authors._id: 68390da8f527444e97c4ab9b
paper_authors.name: David Cox
paper_authors._id: 68390da8f527444e97c4ab9c
paper_authors.name: Gregory Wornell
paper_authors._id: 68390da8f527444e97c4ab9d
paper_authors.name: Wei Lu
paper_authors._id: 68390da8f527444e97c4ab9e
paper_authors.name: Zhang-Wei Hong
paper_authors._id: 68390da8f527444e97c4ab9f
paper_authors.name: Chuang Gan
paper_title: Satori-SWE: Evolutionary Test-Time Scaling for Sample-Efficient Software
    Engineering
paper_summary: Language models (LMs) perform well on standardized coding benchmarks but
  struggle with real-world software engineering tasks such as resolving GitHub
  issues in SWE-Bench, especially when model parameters are less than 100B. While
  smaller models are preferable in practice due to their lower computational
  cost, improving their performance remains challenging. Existing approaches
  primarily rely on supervised fine-tuning (SFT) with high-quality data, which is
  expensive to curate at scale. An alternative is test-time scaling: generating
  multiple outputs, scoring them using a verifier, and selecting the best one.
  Although effective, this strategy often requires excessive sampling and costly
  scoring, limiting its practical application. We propose Evolutionary Test-Time
  Scaling (EvoScale), a sample-efficient method that treats generation as an
  evolutionary process. By iteratively refining outputs via selection and
  mutation, EvoScale shifts the output distribution toward higher-scoring
  regions, reducing the number of samples needed to find correct solutions. To
  reduce the overhead from repeatedly sampling and selection, we train the model
  to self-evolve using reinforcement learning (RL). Rather than relying on
  external verifiers at inference time, the model learns to self-improve the
  scores of its own generations across iterations. Evaluated on
  SWE-Bench-Verified, EvoScale enables our 32B model, Satori-SWE-32B, to match or
  exceed the performance of models with over 100B parameters while using a few
  samples. Code, data, and models will be fully open-sourced.
paper_ai_summary: EvoScale, an evolutionary and reinforcement learning-based method, enhances small language models' performance on real-world software engineering tasks by iteratively improving and refining outputs.
paper_ai_keywords: supervised fine-tuning test-time scaling Evolutionary Test-Time Scaling EvoScale reinforcement learning SWE-Bench-Verified