Afterburner: Reinforcement Learning Facilitates Self-Improving Code
  Efficiency Optimization
Large Language Models (LLMs) generate functionally correct solutions but
often fall short in code efficiency, a critical bottleneck for real-world
deployment. In this paper, we introduce a novel test-time iterative
optimization framework to address this, employing a closed-loop system where
LLMs iteratively refine code based on empirical performance feedback from an
execution sandbox. We explore three training strategies: Supervised Fine-Tuning
(SFT), Direct Preference Optimization (DPO), and Group Relative Policy
Optimization~(GRPO). Experiments on our Venus dataset and the APPS benchmark
show that SFT and DPO rapidly saturate in efficiency gains. In contrast, GRPO,
using reinforcement learning (RL) with execution feedback, continuously
optimizes code performance, significantly boosting both pass@1 (from 47% to
62%) and the likelihood of outperforming human submissions in efficiency (from
31% to 45%). Our work demonstrates effective test-time code efficiency
improvement and critically reveals the power of RL in teaching LLMs to truly
self-improve code efficiency.
68397697a14020a996c30112
Mingzhe Du
claimed_verified
2025-06-07T05:49:51.005Z
61711f02e0b1ddb56eb9b526
/avatars/3e2fdf774f5bc1f73b450486d6da42d4.svg
Mingzhe Du
user
Elfsong
68397697a14020a996c30113
Luu Tuan Tuan
68397697a14020a996c30114
Yue Liu
68397697a14020a996c30115
Yuhao Qing
68397697a14020a996c30116
Dong Huang
68397697a14020a996c30117
Xinyi He
68397697a14020a996c30118
Qian Liu
68397697a14020a996c30119
Zejun Ma
68397697a14020a996c3011a
See-kiong Ng
Afterburner: Reinforcement Learning Facilitates Self-Improving Code
  Efficiency Optimization
Large Language Models (LLMs) generate functionally correct solutions but
often fall short in code efficiency, a critical bottleneck for real-world
deployment. In this paper, we introduce a novel test-time iterative
optimization framework to address this, employing a closed-loop system where
LLMs iteratively refine code based on empirical performance feedback from an
execution sandbox. We explore three training strategies: Supervised Fine-Tuning
(SFT), Direct Preference Optimization (DPO), and Group Relative Policy
Optimization~(GRPO). Experiments on our Venus dataset and the APPS benchmark
show that SFT and DPO rapidly saturate in efficiency gains. In contrast, GRPO,
using reinforcement learning (RL) with execution feedback, continuously
optimizes code performance, significantly boosting both pass@1 (from 47% to
62%) and the likelihood of outperforming human submissions in efficiency (from
31% to 45%). Our work demonstrates effective test-time code efficiency
improvement and critically reveals the power of RL in teaching LLMs to truly
self-improve code efficiency.
A novel test-time iterative optimization framework using reinforcement learning continuously enhances code efficiency generated by large language models.
Large Language Models iterative optimization closed-loop system empirical performance feedback Supervised Fine-Tuning Direct Preference Optimization Group Relative Policy Optimization reinforcement learning pass@1 Venus dataset APPS benchmark