title: ACECODER: Acing Coder RL via Automated Test-Case Synthesis
summary: Most progress in recent coder models has been driven by supervised
  fine-tuning (SFT), while the potential of reinforcement learning (RL) remains
  largely unexplored, primarily due to the lack of reliable reward data/model in
  the code domain. In this paper, we address this challenge by leveraging
  automated large-scale test-case synthesis to enhance code model training.
  Specifically, we design a pipeline that generates extensive (question,
  test-cases) pairs from existing code data. Using these test cases, we construct
  preference pairs based on pass rates over sampled programs to train reward
  models with Bradley-Terry loss. It shows an average of 10-point improvement for
  Llama-3.1-8B-Ins and 5-point improvement for Qwen2.5-Coder-7B-Ins through
  best-of-32 sampling, making the 7B model on par with 236B DeepSeek-V2.5.
  Furthermore, we conduct reinforcement learning with both reward models and
  test-case pass rewards, leading to consistent improvements across HumanEval,
  MBPP, BigCodeBench, and LiveCodeBench (V4). Notably, we follow the R1-style
  training to start from Qwen2.5-Coder-base directly and show that our RL
  training can improve model on HumanEval-plus by over 25\% and MBPP-plus by 6\%
  for merely 80 optimization steps. We believe our results highlight the huge
  potential of reinforcement learning in coder models.
paper_authors._id: 67a2d995c97974764a8c294c
paper_authors.name: Huaye Zeng
paper_authors._id: 67a2d995c97974764a8c294d
paper_authors.name: Dongfu Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-05T10:14:14.136Z
paper_authors._id: 62567c86d444a9b5a0ec51c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62567c86d444a9b5a0ec51c1/1vXJf2uGztPcXpkwyTBr6.png
paper_authors.fullname: Dongfu Jiang
paper_authors.type: user
paper_authors.user: DongfuJiang
paper_authors._id: 67a2d995c97974764a8c294e
paper_authors.name: Haozhe Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-15T08:01:56.171Z
paper_authors._id: 65bf52f0259bc6caeb74f8bf
paper_authors.avatarUrl: /avatars/b38392e954466df784a5760ded5df804.svg
paper_authors.fullname: Haozhe Wang
paper_authors.type: user
paper_authors.user: JasperHaozhe
paper_authors._id: 67a2d995c97974764a8c294f
paper_authors.name: Ping Nie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-05T10:18:51.897Z
paper_authors._id: 65358802a920f38780b3248a
paper_authors.avatarUrl: /avatars/9415510b598079973c2b0436ad12db9c.svg
paper_authors.fullname: Ping Nie
paper_authors.type: user
paper_authors.user: pingnieuk
paper_authors._id: 67a2d995c97974764a8c2950
paper_authors.name: Xiaotong Chen
paper_authors._id: 67a2d995c97974764a8c2951
paper_authors.name: Wenhu Chen
paper_title: ACECODER: Acing Coder RL via Automated Test-Case Synthesis
paper_summary: Most progress in recent coder models has been driven by supervised
  fine-tuning (SFT), while the potential of reinforcement learning (RL) remains
  largely unexplored, primarily due to the lack of reliable reward data/model in
  the code domain. In this paper, we address this challenge by leveraging
  automated large-scale test-case synthesis to enhance code model training.
  Specifically, we design a pipeline that generates extensive (question,
  test-cases) pairs from existing code data. Using these test cases, we construct
  preference pairs based on pass rates over sampled programs to train reward
  models with Bradley-Terry loss. It shows an average of 10-point improvement for
  Llama-3.1-8B-Ins and 5-point improvement for Qwen2.5-Coder-7B-Ins through
  best-of-32 sampling, making the 7B model on par with 236B DeepSeek-V2.5.
  Furthermore, we conduct reinforcement learning with both reward models and
  test-case pass rewards, leading to consistent improvements across HumanEval,
  MBPP, BigCodeBench, and LiveCodeBench (V4). Notably, we follow the R1-style
  training to start from Qwen2.5-Coder-base directly and show that our RL
  training can improve model on HumanEval-plus by over 25\% and MBPP-plus by 6\%
  for merely 80 optimization steps. We believe our results highlight the huge
  potential of reinforcement learning in coder models.
paper_ai_summary: Automated test-case synthesis improves reward models and reinforces coding models, demonstrating significant performance gains.
paper_ai_keywords: supervised fine-tuning reinforcement learning automated large-scale test-case synthesis preference pairs Bradley-Terry loss reward models Llama-3.1-8B-Ins Qwen2.5-Coder-7B-Ins DeepSeek-V2.5 best-of-32 sampling HumanEval MBPP BigCodeBench LiveCodeBench R1-style training HumanEval-plus MBPP-plus