title: Iterative Self-Training for Code Generation via Reinforced Re-Ranking
summary: Generating high-quality code that solves complex programming tasks is
  challenging, especially with current decoder-based models that produce highly
  stochastic outputs. In code generation, even minor errors can easily break the
  entire solution. Leveraging multiple sampled solutions can significantly
  improve the overall output quality.
    One effective way to enhance code generation is by pairing a code generation
  model with a reranker model, which selects the best solution from the generated
  samples. We propose a novel iterative self-training approach for self-training
  reranker models using Proximal Policy Optimization (PPO), aimed at improving
  both reranking accuracy and the overall code generation process. Unlike
  traditional PPO approaches, where the focus is on optimizing a generative model
  with a reward model, our approach emphasizes the development of a robust
  reward/reranking model. This model improves the quality of generated code
  through reranking and addresses problems and errors that the reward model might
  overlook during PPO alignment with the reranker. Our method iteratively refines
  the training dataset by re-evaluating outputs, identifying high-scoring
  negative examples, and incorporating them into the training loop, that boosting
  model performance.
    Our evaluation on the MultiPL-E dataset demonstrates that our 13.4B parameter
  model outperforms a 33B model in code generation quality while being three
  times faster. Moreover, it achieves performance comparable to GPT-4 and
  surpasses it in one programming language.
paper_authors._id: 67fe391f684ac7f04ce21df7
paper_authors.name: Nikita Sorokin
paper_authors._id: 67fe391f684ac7f04ce21df8
paper_authors.name: Ivan Sedykh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-16T09:15:30.948Z
paper_authors._id: 610e8c12119bebecb4d807b6
paper_authors.avatarUrl: /avatars/7230b1584ec45585c12eb5703fd80ff3.svg
paper_authors.fullname: Ivan Sedykh
paper_authors.type: user
paper_authors.user: idsedykh
paper_authors._id: 67fe391f684ac7f04ce21df9
paper_authors.name: Valentin Malykh
paper_title: Iterative Self-Training for Code Generation via Reinforced Re-Ranking
paper_summary: Generating high-quality code that solves complex programming tasks is
  challenging, especially with current decoder-based models that produce highly
  stochastic outputs. In code generation, even minor errors can easily break the
  entire solution. Leveraging multiple sampled solutions can significantly
  improve the overall output quality.
    One effective way to enhance code generation is by pairing a code generation
  model with a reranker model, which selects the best solution from the generated
  samples. We propose a novel iterative self-training approach for self-training
  reranker models using Proximal Policy Optimization (PPO), aimed at improving
  both reranking accuracy and the overall code generation process. Unlike
  traditional PPO approaches, where the focus is on optimizing a generative model
  with a reward model, our approach emphasizes the development of a robust
  reward/reranking model. This model improves the quality of generated code
  through reranking and addresses problems and errors that the reward model might
  overlook during PPO alignment with the reranker. Our method iteratively refines
  the training dataset by re-evaluating outputs, identifying high-scoring
  negative examples, and incorporating them into the training loop, that boosting
  model performance.
    Our evaluation on the MultiPL-E dataset demonstrates that our 13.4B parameter
  model outperforms a 33B model in code generation quality while being three
  times faster. Moreover, it achieves performance comparable to GPT-4 and
  surpasses it in one programming language.
paper_ai_summary: An iterative self-training approach using Proximal Policy Optimization enhances reranker models for code generation, improving accuracy and quality compared to larger models.
paper_ai_keywords: decoder-based models code generation reranker model Proximal Policy Optimization reranking accuracy MultiPL-E dataset GPT-4