Prior Prompt Engineering for Reinforcement Fine-Tuning
This paper investigates prior prompt engineering (pPE) in the context of
reinforcement fine-tuning (RFT), where language models (LMs) are incentivized
to exhibit behaviors that maximize performance through reward signals. While
existing RFT research has primarily focused on algorithms, reward shaping, and
data curation, the design of the prior prompt--the instructions prepended to
queries during training to elicit behaviors such as step-by-step
reasoning--remains underexplored. We investigate whether different pPE
approaches can guide LMs to internalize distinct behaviors after RFT. Inspired
by inference-time prompt engineering (iPE), we translate five representative
iPE strategies--reasoning, planning, code-based reasoning, knowledge recall,
and null-example utilization--into corresponding pPE approaches. We experiment
with Qwen2.5-7B using each of the pPE approaches, then evaluate performance on
in-domain and out-of-domain benchmarks (e.g., AIME2024, HumanEval+, and
GPQA-Diamond). Our results show that all pPE-trained models surpass their
iPE-prompted counterparts, with the null-example pPE approach achieving the
largest average performance gain and the highest improvement on AIME2024 and
GPQA-Diamond, surpassing the commonly used reasoning approach. Furthermore, by
adapting a behavior-classification framework, we demonstrate that different pPE
strategies instill distinct behavioral styles in the resulting models. These
findings position pPE as a powerful yet understudied axis for RFT.
682ee9d0498a5aaed2692828
Pittawat Taveekitworachai
extracted_confirmed
2025-05-22T10:48:05.038Z
615313b0793ef66b3324da1f
https://cdn-avatars.huggingface.co/v1/production/uploads/615313b0793ef66b3324da1f/VyJniD3dxbV5a2CMgVVQ2.jpeg
Pittawat Taveekitworachai
user
pittawat
682ee9d0498a5aaed2692829
Potsawee Manakul
682ee9d0498a5aaed269282a
Sarana Nutanong
682ee9d0498a5aaed269282b
Kunat Pipatanakul
extracted_confirmed
2025-07-14T09:08:18.549Z
62d192c2d50433c35eb1b48e
https://cdn-avatars.huggingface.co/v1/production/uploads/62d192c2d50433c35eb1b48e/VjmDu8GOIuLuQNBQdQLLS.png
Kunat Pipatanakul
user
kunato
Prior Prompt Engineering for Reinforcement Fine-Tuning
This paper investigates prior prompt engineering (pPE) in the context of
reinforcement fine-tuning (RFT), where language models (LMs) are incentivized
to exhibit behaviors that maximize performance through reward signals. While
existing RFT research has primarily focused on algorithms, reward shaping, and
data curation, the design of the prior prompt--the instructions prepended to
queries during training to elicit behaviors such as step-by-step
reasoning--remains underexplored. We investigate whether different pPE
approaches can guide LMs to internalize distinct behaviors after RFT. Inspired
by inference-time prompt engineering (iPE), we translate five representative
iPE strategies--reasoning, planning, code-based reasoning, knowledge recall,
and null-example utilization--into corresponding pPE approaches. We experiment
with Qwen2.5-7B using each of the pPE approaches, then evaluate performance on
in-domain and out-of-domain benchmarks (e.g., AIME2024, HumanEval+, and
GPQA-Diamond). Our results show that all pPE-trained models surpass their
iPE-prompted counterparts, with the null-example pPE approach achieving the
largest average performance gain and the highest improvement on AIME2024 and
GPQA-Diamond, surpassing the commonly used reasoning approach. Furthermore, by
adapting a behavior-classification framework, we demonstrate that different pPE
strategies instill distinct behavioral styles in the resulting models. These
findings position pPE as a powerful yet understudied axis for RFT.
Prior prompt engineering is investigated as a means to guide language models to internalize distinct behaviors through reinforcement fine-tuning, showing significant performance gains over inference-time prompt engineering.
prompt engineering reinforcement fine-tuning language models reward signals step-by-step reasoning Qwen2.5-7B AIME2024 HumanEval+ GPQA-Diamond behavior-classification framework