title: Process-based Self-Rewarding Language Models
summary: Large Language Models have demonstrated outstanding performance across
  various downstream tasks and have been widely applied in multiple scenarios.
  Human-annotated preference data is used for training to further improve LLMs'
  performance, which is constrained by the upper limit of human performance.
  Therefore, Self-Rewarding method has been proposed, where LLMs generate
  training data by rewarding their own outputs. However, the existing
  self-rewarding paradigm is not effective in mathematical reasoning scenarios
  and may even lead to a decline in performance. In this work, we propose the
  Process-based Self-Rewarding pipeline for language models, which introduces
  long-thought reasoning, step-wise LLM-as-a-Judge, and step-wise preference
  optimization within the self-rewarding paradigm. Our new paradigm successfully
  enhances the performance of LLMs on multiple mathematical reasoning benchmarks
  through iterative Process-based Self-Rewarding, demonstrating the immense
  potential of self-rewarding to achieve LLM reasoning that may surpass human
  capabilities.
paper_authors._id: 67c991abe7dba9cdf9f41e67
paper_authors.name: Shimao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-06T19:50:45.146Z
paper_authors._id: 65080dc63fc966d1bbba485d
paper_authors.avatarUrl: /avatars/347890233f2316e7f7a04d652b2378bb.svg
paper_authors.fullname: Shimao Zhang
paper_authors.type: user
paper_authors.user: Shimao-Zhang
paper_authors._id: 67c991abe7dba9cdf9f41e68
paper_authors.name: Xiao Liu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-06T12:14:52.016Z
paper_authors._id: 63fb6e281b4b1bd4e7ffc5be
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1677422062937-noauth.jpeg
paper_authors.fullname: Xiao Liu
paper_authors.type: user
paper_authors.user: lx865712528
paper_authors._id: 67c991abe7dba9cdf9f41e69
paper_authors.name: Xin Zhang
paper_authors._id: 67c991abe7dba9cdf9f41e6a
paper_authors.name: Junxiao Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T07:56:21.963Z
paper_authors._id: 68356f5db243fb809813a715
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/68356f5db243fb809813a715/grhHvANfDRp75rMJxWlQo.jpeg
paper_authors.fullname: LiuJunxiao
paper_authors.type: user
paper_authors.user: master-lan
paper_authors._id: 67c991abe7dba9cdf9f41e6b
paper_authors.name: Zheheng Luo
paper_authors._id: 67c991abe7dba9cdf9f41e6c
paper_authors.name: Shujian Huang
paper_authors._id: 67c991abe7dba9cdf9f41e6d
paper_authors.name: Yeyun Gong
paper_title: Process-based Self-Rewarding Language Models
paper_summary: Large Language Models have demonstrated outstanding performance across
  various downstream tasks and have been widely applied in multiple scenarios.
  Human-annotated preference data is used for training to further improve LLMs'
  performance, which is constrained by the upper limit of human performance.
  Therefore, Self-Rewarding method has been proposed, where LLMs generate
  training data by rewarding their own outputs. However, the existing
  self-rewarding paradigm is not effective in mathematical reasoning scenarios
  and may even lead to a decline in performance. In this work, we propose the
  Process-based Self-Rewarding pipeline for language models, which introduces
  long-thought reasoning, step-wise LLM-as-a-Judge, and step-wise preference
  optimization within the self-rewarding paradigm. Our new paradigm successfully
  enhances the performance of LLMs on multiple mathematical reasoning benchmarks
  through iterative Process-based Self-Rewarding, demonstrating the immense
  potential of self-rewarding to achieve LLM reasoning that may surpass human
  capabilities.
paper_ai_summary: A Process-based Self-Rewarding pipeline enhances LLMs' mathematical reasoning by iteratively evaluating and optimizing their own outputs.
paper_ai_keywords: Large Language Models self-rewarding LLM-as-a-Judge step-wise preference optimization mathematical reasoning Process-based Self-Rewarding