title: Synthetic Data RL: Task Definition Is All You Need
summary: Reinforcement learning (RL) is a powerful way to adapt foundation models to
  specialized tasks, but its reliance on large-scale human-labeled data limits
  broad adoption. We introduce Synthetic Data RL, a simple and general framework
  that reinforcement fine-tunes models using only synthetic data generated from a
  task definition. Our method first generates question and answer pairs from the
  task definition and retrieved documents, then adapts the difficulty of the
  question based on model solvability, and selects questions using the average
  pass rate of the model across samples for RL training. On Qwen-2.5-7B, our
  method achieves a 29.2% absolute improvement over the base model on GSM8K (+2.9
  pp vs. instruction-tuned, +6.6 pp vs. Self-Instruct), 8.7% on MATH, 13.1% on
  GPQA (+7.0 pp vs. SynthLLM), 8.9% on MedQA, 17.7% on CQA (law) and 13.7% on CFA
  (finance). It surpasses supervised fine-tuning under the same data budget and
  nearly matches RL with full human data across datasets (e.g., +17.2 pp on
  GSM8K). Adding 100 human demonstrations improves the performance of GSM8K only
  by 0.4 pp, showing a limited added value. By reducing human data annotation,
  Synthetic Data RL enables scalable and efficient RL-based model adaptation.
  Code and demos are available at https://github.com/gydpku/Data_Synthesis_RL/.
paper_authors._id: 6833e65bf9ae3819ea4c568e
paper_authors.name: Yiduo Guo
paper_authors._id: 6833e65bf9ae3819ea4c568f
paper_authors.name: Zhen Guo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:09:28.399Z
paper_authors._id: 638e4e66629b4d0a62ce1bf3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638e4e66629b4d0a62ce1bf3/s7uQ2qmee2CaXfjZDOovZ.jpeg
paper_authors.fullname: Zhen Guo
paper_authors.type: user
paper_authors.user: zguo0525
paper_authors._id: 6833e65bf9ae3819ea4c5690
paper_authors.name: Chuanwei Huang
paper_authors._id: 6833e65bf9ae3819ea4c5691
paper_authors.name: Zi-Ang Wang
paper_authors._id: 6833e65bf9ae3819ea4c5692
paper_authors.name: Zekai Zhang
paper_authors._id: 6833e65bf9ae3819ea4c5693
paper_authors.name: Haofei Yu
paper_authors._id: 6833e65bf9ae3819ea4c5694
paper_authors.name: Huishuai Zhang
paper_authors._id: 6833e65bf9ae3819ea4c5695
paper_authors.name: Yikang Shen
paper_title: Synthetic Data RL: Task Definition Is All You Need
paper_summary: Reinforcement learning (RL) is a powerful way to adapt foundation models to
  specialized tasks, but its reliance on large-scale human-labeled data limits
  broad adoption. We introduce Synthetic Data RL, a simple and general framework
  that reinforcement fine-tunes models using only synthetic data generated from a
  task definition. Our method first generates question and answer pairs from the
  task definition and retrieved documents, then adapts the difficulty of the
  question based on model solvability, and selects questions using the average
  pass rate of the model across samples for RL training. On Qwen-2.5-7B, our
  method achieves a 29.2% absolute improvement over the base model on GSM8K (+2.9
  pp vs. instruction-tuned, +6.6 pp vs. Self-Instruct), 8.7% on MATH, 13.1% on
  GPQA (+7.0 pp vs. SynthLLM), 8.9% on MedQA, 17.7% on CQA (law) and 13.7% on CFA
  (finance). It surpasses supervised fine-tuning under the same data budget and
  nearly matches RL with full human data across datasets (e.g., +17.2 pp on
  GSM8K). Adding 100 human demonstrations improves the performance of GSM8K only
  by 0.4 pp, showing a limited added value. By reducing human data annotation,
  Synthetic Data RL enables scalable and efficient RL-based model adaptation.
  Code and demos are available at https://github.com/gydpku/Data_Synthesis_RL/.
paper_ai_summary: Synthetic Data RL enhances foundation models through reinforcement learning using only synthetic data, achieving performance comparable to models trained with full human-labeled data.
paper_ai_keywords: reinforcement learning RL synthetic data reinforcement fine-tuning question and answer pairs model solvability average pass rate data budget supervised fine-tuning