title: Multi-Turn Code Generation Through Single-Step Rewards
summary: We address the problem of code generation from multi-turn execution feedback.
  Existing methods either generate code without feedback or use complex,
  hierarchical reinforcement learning to optimize multi-turn rewards. We propose
  a simple yet scalable approach, muCode, that solves multi-turn code
  generation using only single-step rewards. Our key insight is that code
  generation is a one-step recoverable MDP, where the correct code can be
  recovered from any intermediate code state in a single turn. muCode
  iteratively trains both a generator to provide code solutions conditioned on
  multi-turn execution feedback and a verifier to score the newly generated code.
  Experimental evaluations show that our approach achieves significant
  improvements over the state-of-the-art baselines. We provide analysis of the
  design choices of the reward models and policy, and show the efficacy of
  muCode at utilizing the execution feedback. Our code is available at
  https://github.com/portal-cornell/muCode.
paper_authors._id: 67c34e3beae05d8f94f800b4
paper_authors.name: Arnav Kumar Jain
paper_authors._id: 67c34e3beae05d8f94f800b5
paper_authors.name: Gonzalo Gonzalez-Pumariega
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-02T20:15:14.593Z
paper_authors._id: 6421d2972143035270db37b9
paper_authors.avatarUrl: /avatars/4fadeafc273d32cf72fe2f12d444c5e8.svg
paper_authors.fullname: Gonzalo Gonzalez
paper_authors.type: user
paper_authors.user: chalo2000
paper_authors._id: 67c34e3beae05d8f94f800b6
paper_authors.name: Wayne Chen
paper_authors._id: 67c34e3beae05d8f94f800b7
paper_authors.name: Alexander M Rush
paper_authors._id: 67c34e3beae05d8f94f800b8
paper_authors.name: Wenting Zhao
paper_authors._id: 67c34e3beae05d8f94f800b9
paper_authors.name: Sanjiban Choudhury
paper_title: Multi-Turn Code Generation Through Single-Step Rewards
paper_summary: We address the problem of code generation from multi-turn execution feedback.
  Existing methods either generate code without feedback or use complex,
  hierarchical reinforcement learning to optimize multi-turn rewards. We propose
  a simple yet scalable approach, muCode, that solves multi-turn code
  generation using only single-step rewards. Our key insight is that code
  generation is a one-step recoverable MDP, where the correct code can be
  recovered from any intermediate code state in a single turn. muCode
  iteratively trains both a generator to provide code solutions conditioned on
  multi-turn execution feedback and a verifier to score the newly generated code.
  Experimental evaluations show that our approach achieves significant
  improvements over the state-of-the-art baselines. We provide analysis of the
  design choices of the reward models and policy, and show the efficacy of
  muCode at utilizing the execution feedback. Our code is available at
  https://github.com/portal-cornell/muCode.
paper_ai_keywords: MDP $\mu$Code generator verifier reward models policy