RLTF: Reinforcement Learning from Unit Test Feedback
The goal of program synthesis, or code generation, is to generate executable
code based on given descriptions. Recently, there has been an increasing number
of studies employing reinforcement learning (RL) to improve the performance of
large language models (LLMs) for code. However, these RL methods have only used
offline frameworks, limiting their exploration of new sample spaces.
Additionally, current approaches that utilize unit test signals are rather
simple, not accounting for specific error locations within the code. To address
these issues, we proposed RLTF, i.e., Reinforcement Learning from Unit Test
Feedback, a novel online RL framework with unit test feedback of
multi-granularity for refining code LLMs. Our approach generates data in
real-time during training and simultaneously utilizes fine-grained feedback
signals to guide the model towards producing higher-quality code. Extensive
experiments show that RLTF achieves state-of-the-art performance on the APPS
and the MBPP benchmarks. Our code can be found at:
https://github.com/Zyq-scut/RLTF.
64acb35058bd9e9cc2c1b2fc
Jiate Liu
64acb35058bd9e9cc2c1b2fd
Yiqin Zhu
claimed_verified
2023-07-12T07:29:49.317Z
63e3924c19b1b962a43e331c
/avatars/b7b741bec01b7479a16e5e17dcef472d.svg
yiqin zhu
user
Zyego
64acb35058bd9e9cc2c1b2fe
Kaiwen Xiao
claimed_verified
2023-07-18T06:44:55.326Z
6441491c5d600fb0951cd872
/avatars/d98892f3b52d87c2328201efa9897110.svg
Kaiwen Xiao
user
loktarxiao
64acb35058bd9e9cc2c1b2ff
Qiang Fu
64acb35058bd9e9cc2c1b300
Xiao Han
64acb35058bd9e9cc2c1b301
Wei Yang
64acb35058bd9e9cc2c1b302
Deheng Ye
RLTF: Reinforcement Learning from Unit Test Feedback
The goal of program synthesis, or code generation, is to generate executable
code based on given descriptions. Recently, there has been an increasing number
of studies employing reinforcement learning (RL) to improve the performance of
large language models (LLMs) for code. However, these RL methods have only used
offline frameworks, limiting their exploration of new sample spaces.
Additionally, current approaches that utilize unit test signals are rather
simple, not accounting for specific error locations within the code. To address
these issues, we proposed RLTF, i.e., Reinforcement Learning from Unit Test
Feedback, a novel online RL framework with unit test feedback of
multi-granularity for refining code LLMs. Our approach generates data in
real-time during training and simultaneously utilizes fine-grained feedback
signals to guide the model towards producing higher-quality code. Extensive
experiments show that RLTF achieves state-of-the-art performance on the APPS
and the MBPP benchmarks. Our code can be found at:
https://github.com/Zyq-scut/RLTF.
A new reinforcement learning framework using multi-granularity unit test feedback enhances code generation with large language models, achieving superior performance on benchmarks.
reinforcement learning large language models offline frameworks unit test feedback RLTF real-time training fine-grained feedback code generation APPS MBPP benchmarks