title: Prior Prompt Engineering for Reinforcement Fine-Tuning
summary: This paper investigates prior prompt engineering (pPE) in the context of
  reinforcement fine-tuning (RFT), where language models (LMs) are incentivized
  to exhibit behaviors that maximize performance through reward signals. While
  existing RFT research has primarily focused on algorithms, reward shaping, and
  data curation, the design of the prior prompt--the instructions prepended to
  queries during training to elicit behaviors such as step-by-step
  reasoning--remains underexplored. We investigate whether different pPE
  approaches can guide LMs to internalize distinct behaviors after RFT. Inspired
  by inference-time prompt engineering (iPE), we translate five representative
  iPE strategies--reasoning, planning, code-based reasoning, knowledge recall,
  and null-example utilization--into corresponding pPE approaches. We experiment
  with Qwen2.5-7B using each of the pPE approaches, then evaluate performance on
  in-domain and out-of-domain benchmarks (e.g., AIME2024, HumanEval+, and
  GPQA-Diamond). Our results show that all pPE-trained models surpass their
  iPE-prompted counterparts, with the null-example pPE approach achieving the
  largest average performance gain and the highest improvement on AIME2024 and
  GPQA-Diamond, surpassing the commonly used reasoning approach. Furthermore, by
  adapting a behavior-classification framework, we demonstrate that different pPE
  strategies instill distinct behavioral styles in the resulting models. These
  findings position pPE as a powerful yet understudied axis for RFT.
paper_authors._id: 682ee9d0498a5aaed2692828
paper_authors.name: Pittawat Taveekitworachai
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-05-22T10:48:05.038Z
paper_authors._id: 615313b0793ef66b3324da1f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/615313b0793ef66b3324da1f/VyJniD3dxbV5a2CMgVVQ2.jpeg
paper_authors.fullname: Pittawat Taveekitworachai
paper_authors.type: user
paper_authors.user: pittawat
paper_authors._id: 682ee9d0498a5aaed2692829
paper_authors.name: Potsawee Manakul
paper_authors._id: 682ee9d0498a5aaed269282a
paper_authors.name: Sarana Nutanong
paper_authors._id: 682ee9d0498a5aaed269282b
paper_authors.name: Kunat Pipatanakul
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-05-22T09:09:37.412Z
paper_authors._id: 62d192c2d50433c35eb1b48e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62d192c2d50433c35eb1b48e/VjmDu8GOIuLuQNBQdQLLS.png
paper_authors.fullname: Kunat Pipatanakul
paper_authors.type: user
paper_authors.user: kunato
paper_title: Prior Prompt Engineering for Reinforcement Fine-Tuning
paper_summary: This paper investigates prior prompt engineering (pPE) in the context of
  reinforcement fine-tuning (RFT), where language models (LMs) are incentivized
  to exhibit behaviors that maximize performance through reward signals. While
  existing RFT research has primarily focused on algorithms, reward shaping, and
  data curation, the design of the prior prompt--the instructions prepended to
  queries during training to elicit behaviors such as step-by-step
  reasoning--remains underexplored. We investigate whether different pPE
  approaches can guide LMs to internalize distinct behaviors after RFT. Inspired
  by inference-time prompt engineering (iPE), we translate five representative
  iPE strategies--reasoning, planning, code-based reasoning, knowledge recall,
  and null-example utilization--into corresponding pPE approaches. We experiment
  with Qwen2.5-7B using each of the pPE approaches, then evaluate performance on
  in-domain and out-of-domain benchmarks (e.g., AIME2024, HumanEval+, and
  GPQA-Diamond). Our results show that all pPE-trained models surpass their
  iPE-prompted counterparts, with the null-example pPE approach achieving the
  largest average performance gain and the highest improvement on AIME2024 and
  GPQA-Diamond, surpassing the commonly used reasoning approach. Furthermore, by
  adapting a behavior-classification framework, we demonstrate that different pPE
  strategies instill distinct behavioral styles in the resulting models. These
  findings position pPE as a powerful yet understudied axis for RFT.
paper_ai_summary: Prior prompt engineering is investigated as a means to guide language models to internalize distinct behaviors through reinforcement fine-tuning, showing significant performance gains over inference-time prompt engineering.
paper_ai_keywords: prompt engineering reinforcement fine-tuning language models reward signals step-by-step reasoning Qwen2.5-7B AIME2024 HumanEval+ GPQA-Diamond behavior-classification framework