title: StepCoder: Improve Code Generation with Reinforcement Learning from
    Compiler Feedback
summary: The advancement of large language models (LLMs) has significantly propelled
  the field of code generation. Previous work integrated reinforcement learning
  (RL) with compiler feedback for exploring the output space of LLMs to enhance
  code generation quality. However, the lengthy code generated by LLMs in
  response to complex human requirements makes RL exploration a challenge. Also,
  since the unit tests may not cover the complicated code, optimizing LLMs by
  using these unexecuted code snippets is ineffective. To tackle these
  challenges, we introduce StepCoder, a novel RL framework for code generation,
  consisting of two main components: CCCS addresses the exploration challenge by
  breaking the long sequences code generation task into a Curriculum of Code
  Completion Subtasks, while FGO only optimizes the model by masking the
  unexecuted code segments to provide Fine-Grained Optimization. In addition, we
  furthermore construct the APPS+ dataset for RL training, which is manually
  verified to ensure the correctness of unit tests. Experimental results show
  that our method improves the ability to explore the output space and
  outperforms state-of-the-art approaches in corresponding benchmarks.
paper_authors._id: 65c04ff5c63d6a8d7f234771
paper_authors.name: Shihan Dou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-05T12:01:46.974Z
paper_authors._id: 64ae0b6ccf90fe27556cf56e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/uVXJd3TBssT_Ug4yAl3c9.jpeg
paper_authors.fullname: Shihan Dou
paper_authors.type: user
paper_authors.user: Ablustrund
paper_authors._id: 65c04ff5c63d6a8d7f234772
paper_authors.name: Yan Liu
paper_authors._id: 65c04ff5c63d6a8d7f234773
paper_authors.name: Haoxiang Jia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-05T12:02:33.315Z
paper_authors._id: 644606aee1fd8d65b27e2744
paper_authors.avatarUrl: /avatars/08f742765f66114181a0be817c7fea5f.svg
paper_authors.fullname: Haoxiang Jia
paper_authors.type: user
paper_authors.user: jiatastic
paper_authors._id: 65c04ff5c63d6a8d7f234774
paper_authors.name: Limao Xiong
paper_authors._id: 65c04ff5c63d6a8d7f234775
paper_authors.name: Enyu Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-05T12:03:11.210Z
paper_authors._id: 64c28861c0126fdf8a285c92
paper_authors.avatarUrl: /avatars/409c914c827282f4276d3500561fabf3.svg
paper_authors.fullname: enyu
paper_authors.type: user
paper_authors.user: zhouenyu04
paper_authors._id: 65c04ff5c63d6a8d7f234776
paper_authors.name: Junjie Shan
paper_authors._id: 65c04ff5c63d6a8d7f234777
paper_authors.name: Caishuang Huang
paper_authors._id: 65c04ff5c63d6a8d7f234778
paper_authors.name: Wei Shen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-28T08:03:23.391Z
paper_authors._id: 62308ae8be7833fa98620372
paper_authors.avatarUrl: /avatars/220ed57376202c61ec1afbf5a4f45b67.svg
paper_authors.fullname: weishen
paper_authors.type: user
paper_authors.user: fakerbaby
paper_authors._id: 65c04ff5c63d6a8d7f234779
paper_authors.name: Xiaoran Fan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-05T12:03:47.870Z
paper_authors._id: 6424fa5f988b8b19edba8ed4
paper_authors.avatarUrl: /avatars/cff4f1535632e7c8abc33fe7b862f60d.svg
paper_authors.fullname: Xiaoran Fan
paper_authors.type: user
paper_authors.user: cnxup
paper_authors._id: 65c04ff5c63d6a8d7f23477a
paper_authors.name: Zhiheng Xi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-05T12:03:55.868Z
paper_authors._id: 653a6e5cae155b92bae77b74
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/653a6e5cae155b92bae77b74/TA5FWKAUsB249ux4MzD_R.jpeg
paper_authors.fullname: Zhiheng Xi
paper_authors.type: user
paper_authors.user: WooooDyy
paper_authors._id: 65c04ff5c63d6a8d7f23477b
paper_authors.name: Yuhao Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-05T12:04:03.901Z
paper_authors._id: 646dbfcaf813cfe153ef2964
paper_authors.avatarUrl: /avatars/8e73f91b35ac0d11671fd7399a43e723.svg
paper_authors.fullname: Yuhao Zhou
paper_authors.type: user
paper_authors.user: YuhaoZhou
paper_authors._id: 65c04ff5c63d6a8d7f23477c
paper_authors.name: Tao Ji
paper_authors._id: 65c04ff5c63d6a8d7f23477d
paper_authors.name: Rui Zheng
paper_authors._id: 65c04ff5c63d6a8d7f23477e
paper_authors.name: Qi Zhang
paper_authors._id: 65c04ff5c63d6a8d7f23477f
paper_authors.name: Xuanjing Huang
paper_authors._id: 65c04ff5c63d6a8d7f234780
paper_authors.name: Tao Gui
paper_title: StepCoder: Improve Code Generation with Reinforcement Learning from
    Compiler Feedback
paper_summary: The advancement of large language models (LLMs) has significantly propelled
  the field of code generation. Previous work integrated reinforcement learning
  (RL) with compiler feedback for exploring the output space of LLMs to enhance
  code generation quality. However, the lengthy code generated by LLMs in
  response to complex human requirements makes RL exploration a challenge. Also,
  since the unit tests may not cover the complicated code, optimizing LLMs by
  using these unexecuted code snippets is ineffective. To tackle these
  challenges, we introduce StepCoder, a novel RL framework for code generation,
  consisting of two main components: CCCS addresses the exploration challenge by
  breaking the long sequences code generation task into a Curriculum of Code
  Completion Subtasks, while FGO only optimizes the model by masking the
  unexecuted code segments to provide Fine-Grained Optimization. In addition, we
  furthermore construct the APPS+ dataset for RL training, which is manually
  verified to ensure the correctness of unit tests. Experimental results show
  that our method improves the ability to explore the output space and
  outperforms state-of-the-art approaches in corresponding benchmarks.
paper_ai_keywords: large language models (LLMs) code generation reinforcement learning (RL) compiler feedback output space code generation quality RL framework Curriculum of Code Completion Subtasks (CCCS) Fine-Grained Optimization (FGO) unexecuted code snippets APPS+ dataset unit tests correctness