title: Surrogate Signals from Format and Length: Reinforcement Learning for
    Solving Mathematical Problems without Ground Truth Answers
summary: Large Language Models have achieved remarkable success in natural language
  processing tasks, with Reinforcement Learning playing a key role in adapting
  them to specific applications. However, obtaining ground truth answers for
  training LLMs in mathematical problem-solving is often challenging, costly, and
  sometimes unfeasible. This research delves into the utilization of format and
  length as surrogate signals to train LLMs for mathematical problem-solving,
  bypassing the need for traditional ground truth answers.Our study shows that a
  reward function centered on format correctness alone can yield performance
  improvements comparable to the standard GRPO algorithm in early phases.
  Recognizing the limitations of format-only rewards in the later phases, we
  incorporate length-based rewards. The resulting GRPO approach, leveraging
  format-length surrogate signals, not only matches but surpasses the performance
  of the standard GRPO algorithm relying on ground truth answers in certain
  scenarios, achieving 40.0\% accuracy on AIME2024 with a 7B base model. Through
  systematic exploration and experimentation, this research not only offers a
  practical solution for training LLMs to solve mathematical problems and
  reducing the dependence on extensive ground truth data collection, but also
  reveals the essence of why our label-free approach succeeds: base model is like
  an excellent student who has already mastered mathematical and logical
  reasoning skills, but performs poorly on the test paper, it simply needs to
  develop good answering habits to achieve outstanding results in exams , in
  other words, to unlock the capabilities it already possesses.
paper_authors._id: 68355784bb7d114755346770
paper_authors.name: Rihui Xin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T13:01:54.708Z
paper_authors._id: 6835378ad0fbc64a8e718c45
paper_authors.avatarUrl: /avatars/ab0553b3114cc4cfad9eaa034af3241d.svg
paper_authors.fullname: xinrihui
paper_authors.type: user
paper_authors.user: xinrihui
paper_authors._id: 68355784bb7d114755346771
paper_authors.name: Han Liu
paper_authors._id: 68355784bb7d114755346772
paper_authors.name: Zecheng Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T13:02:02.524Z
paper_authors._id: 67123ec5b832202bddc6e74a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/M87nMhWm9LXi0MqxODXVK.png
paper_authors.fullname: Zecheng Wang
paper_authors.type: user
paper_authors.user: wangzech
paper_authors._id: 68355784bb7d114755346773
paper_authors.name: Yupeng Zhang
paper_authors._id: 68355784bb7d114755346774
paper_authors.name: Dianbo Sui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T13:02:22.250Z
paper_authors._id: 643e8b572263cdc630f80fee
paper_authors.avatarUrl: /avatars/93b11c28a58c5db71afd44e782493140.svg
paper_authors.fullname: Dianbo Sui
paper_authors.type: user
paper_authors.user: suidianbo
paper_authors._id: 68355784bb7d114755346775
paper_authors.name: Xiaolin Hu
paper_authors._id: 68355784bb7d114755346776
paper_authors.name: Bingning Wang
paper_title: Surrogate Signals from Format and Length: Reinforcement Learning for
    Solving Mathematical Problems without Ground Truth Answers
paper_summary: Large Language Models have achieved remarkable success in natural language
  processing tasks, with Reinforcement Learning playing a key role in adapting
  them to specific applications. However, obtaining ground truth answers for
  training LLMs in mathematical problem-solving is often challenging, costly, and
  sometimes unfeasible. This research delves into the utilization of format and
  length as surrogate signals to train LLMs for mathematical problem-solving,
  bypassing the need for traditional ground truth answers.Our study shows that a
  reward function centered on format correctness alone can yield performance
  improvements comparable to the standard GRPO algorithm in early phases.
  Recognizing the limitations of format-only rewards in the later phases, we
  incorporate length-based rewards. The resulting GRPO approach, leveraging
  format-length surrogate signals, not only matches but surpasses the performance
  of the standard GRPO algorithm relying on ground truth answers in certain
  scenarios, achieving 40.0\% accuracy on AIME2024 with a 7B base model. Through
  systematic exploration and experimentation, this research not only offers a
  practical solution for training LLMs to solve mathematical problems and
  reducing the dependence on extensive ground truth data collection, but also
  reveals the essence of why our label-free approach succeeds: base model is like
  an excellent student who has already mastered mathematical and logical
  reasoning skills, but performs poorly on the test paper, it simply needs to
  develop good answering habits to achieve outstanding results in exams , in
  other words, to unlock the capabilities it already possesses.
paper_ai_summary: The research demonstrates that using format and length as surrogate signals can improve LLMs' performance in mathematical problem-solving, matching or surpassing traditional methods without extensive ground truth data.
paper_ai_keywords: Large Language Models Reinforcement Learning mathematical problem-solving GRPO algorithm format correctness length-based rewards AIME2024