title: Pairwise RM: Perform Best-of-N Sampling with Knockout Tournament
summary: Best-of-N (BoN) sampling, a common strategy for test-time scaling of Large
  Language Models (LLMs), relies on reward models to select the best candidate
  solution from multiple generations. However, traditional reward models often
  assign arbitrary and inconsistent scores, limiting their effectiveness. To
  address this, we propose a Pairwise Reward Model (Pairwise RM) combined with a
  knockout tournament for BoN sampling. Instead of assigning absolute scores,
  given one math problem, Pairwise RM evaluates two candidate solutions'
  correctness simultaneously. This approach eliminates the need for arbitrary
  scoring and enables cross-validation of solutions through parallel comparison.
  In the knockout tournament, Pairwise RM conducts pairwise comparisons between
  candidate solutions and eliminates the incorrect ones iteratively. We construct
  \ourdataset, a large-scale dataset of 443K pairwise comparisons derived from
  NumiaMath and annotated using gemini-1.5-flash, and train the Pairwise
  RM via supervised fine-tuning. Experiments on MATH-500 and the Olympiad Bench
  demonstrate significant improvements over traditional discriminative reward
  models. And a 40\% to 60\% relative improvement is achieved on the top 50\%
  challenging problems.
paper_authors._id: 6791d751f29664a338e7b4c5
paper_authors.name: Yantao Liu
paper_authors._id: 6791d751f29664a338e7b4c6
paper_authors.name: Zijun Yao
paper_authors._id: 6791d751f29664a338e7b4c7
paper_authors.name: Rui Min
paper_authors._id: 6791d751f29664a338e7b4c8
paper_authors.name: Yixin Cao
paper_authors._id: 6791d751f29664a338e7b4c9
paper_authors.name: Lei Hou
paper_authors._id: 6791d751f29664a338e7b4ca
paper_authors.name: Juanzi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-23T09:49:20.111Z
paper_authors._id: 65df8cbc2705d9672f55d1aa
paper_authors.avatarUrl: /avatars/63e46f15bb76bd9d4508fd0f54f39829.svg
paper_authors.fullname: Juanzi Li
paper_authors.type: user
paper_authors.user: juanli
paper_title: Pairwise RM: Perform Best-of-N Sampling with Knockout Tournament
paper_summary: Best-of-N (BoN) sampling, a common strategy for test-time scaling of Large
  Language Models (LLMs), relies on reward models to select the best candidate
  solution from multiple generations. However, traditional reward models often
  assign arbitrary and inconsistent scores, limiting their effectiveness. To
  address this, we propose a Pairwise Reward Model (Pairwise RM) combined with a
  knockout tournament for BoN sampling. Instead of assigning absolute scores,
  given one math problem, Pairwise RM evaluates two candidate solutions'
  correctness simultaneously. This approach eliminates the need for arbitrary
  scoring and enables cross-validation of solutions through parallel comparison.
  In the knockout tournament, Pairwise RM conducts pairwise comparisons between
  candidate solutions and eliminates the incorrect ones iteratively. We construct
  \ourdataset, a large-scale dataset of 443K pairwise comparisons derived from
  NumiaMath and annotated using gemini-1.5-flash, and train the Pairwise
  RM via supervised fine-tuning. Experiments on MATH-500 and the Olympiad Bench
  demonstrate significant improvements over traditional discriminative reward
  models. And a 40\% to 60\% relative improvement is achieved on the top 50\%
  challenging problems.
paper_ai_summary: Pairwise Reward Model combined with a knockout tournament enhances test-time scaling of Large Language Models by accurately comparing candidate solutions through parallel evaluation.
paper_ai_keywords: Best-of-N (BoN) sampling Large Language Models (LLMs) reward models Pairwise Reward Model knockout tournament NumiaMath MATH-500 Olympiad Bench