title: LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level
    Mathematical Reasoning
summary: This paper presents an advanced mathematical problem-solving framework,
  LLaMA-Berry, for enhancing the mathematical reasoning ability of Large Language
  Models (LLMs). The framework combines Monte Carlo Tree Search (MCTS) with
  iterative Self-Refine to optimize the reasoning path and utilizes a pairwise
  reward model to evaluate different paths globally. By leveraging the
  self-critic and rewriting capabilities of LLMs, Self-Refine applied to MCTS
  (SR-MCTS) overcomes the inefficiencies and limitations of conventional
  step-wise and greedy search algorithms by fostering a more efficient
  exploration of solution spaces. Pairwise Preference Reward Model~(PPRM),
  inspired by Reinforcement Learning from Human Feedback (RLHF), is then used to
  model pairwise preferences between solutions, utilizing an Enhanced Borda Count
  (EBC) method to synthesize these preferences into a global ranking score to
  find better answers. This approach addresses the challenges of scoring
  variability and non-independent distributions in mathematical reasoning tasks.
  The framework has been tested on general and advanced benchmarks, showing
  superior performance in terms of search efficiency and problem-solving
  capability compared to existing methods like ToT and rStar, particularly in
  complex Olympiad-level benchmarks, including GPQA, AIME24 and AMC23.
paper_authors._id: 6704ec6a0fbef97683f15843
paper_authors.name: Di Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T09:43:53.492Z
paper_authors._id: 64bce15bafd1e46c5504ad38
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64bce15bafd1e46c5504ad38/bQFX1iFbXEBXcQvUNL811.png
paper_authors.fullname: Di Zhang
paper_authors.type: user
paper_authors.user: di-zhang-fdu
paper_authors._id: 6704ec6a0fbef97683f15844
paper_authors.name: Jianbo Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T09:43:59.714Z
paper_authors._id: 6407d3d3a7bc7c3865addc4b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6407d3d3a7bc7c3865addc4b/NEMP6xShRq-5H_rQ5ganY.jpeg
paper_authors.fullname: Jianbo Wu
paper_authors.type: user
paper_authors.user: jwu323
paper_authors._id: 6704ec6a0fbef97683f15845
paper_authors.name: Jingdi Lei
paper_authors._id: 6704ec6a0fbef97683f15846
paper_authors.name: Tong Che
paper_authors._id: 6704ec6a0fbef97683f15847
paper_authors.name: Jiatong Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-09T11:21:05.725Z
paper_authors._id: 6438e55cb2ea24b52ebc45ec
paper_authors.avatarUrl: /avatars/e13c7398f77e7e0bd5eed03102aa5c36.svg
paper_authors.fullname: Jiatong LI
paper_authors.type: user
paper_authors.user: phenixace
paper_authors._id: 6704ec6a0fbef97683f15848
paper_authors.name: Tong Xie
paper_authors._id: 6704ec6a0fbef97683f15849
paper_authors.name: Xiaoshui Huang
paper_authors._id: 6704ec6a0fbef97683f1584a
paper_authors.name: Shufei Zhang
paper_authors._id: 6704ec6a0fbef97683f1584b
paper_authors.name: Marco Pavone
paper_authors._id: 6704ec6a0fbef97683f1584c
paper_authors.name: Yuqiang Li
paper_authors._id: 6704ec6a0fbef97683f1584d
paper_authors.name: Wanli Ouyang
paper_authors._id: 6704ec6a0fbef97683f1584e
paper_authors.name: Dongzhan Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T09:47:27.102Z
paper_authors._id: 6538b861613fe158bd581e35
paper_authors.avatarUrl: /avatars/6817dbfe903675721fd227058b0a91ac.svg
paper_authors.fullname: Dongzhan Zhou
paper_authors.type: user
paper_authors.user: schrodingers-tiger
paper_title: LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level
    Mathematical Reasoning
paper_summary: This paper presents an advanced mathematical problem-solving framework,
  LLaMA-Berry, for enhancing the mathematical reasoning ability of Large Language
  Models (LLMs). The framework combines Monte Carlo Tree Search (MCTS) with
  iterative Self-Refine to optimize the reasoning path and utilizes a pairwise
  reward model to evaluate different paths globally. By leveraging the
  self-critic and rewriting capabilities of LLMs, Self-Refine applied to MCTS
  (SR-MCTS) overcomes the inefficiencies and limitations of conventional
  step-wise and greedy search algorithms by fostering a more efficient
  exploration of solution spaces. Pairwise Preference Reward Model~(PPRM),
  inspired by Reinforcement Learning from Human Feedback (RLHF), is then used to
  model pairwise preferences between solutions, utilizing an Enhanced Borda Count
  (EBC) method to synthesize these preferences into a global ranking score to
  find better answers. This approach addresses the challenges of scoring
  variability and non-independent distributions in mathematical reasoning tasks.
  The framework has been tested on general and advanced benchmarks, showing
  superior performance in terms of search efficiency and problem-solving
  capability compared to existing methods like ToT and rStar, particularly in
  complex Olympiad-level benchmarks, including GPQA, AIME24 and AMC23.
paper_ai_keywords: Large Language Models (LLMs) Monte Carlo Tree Search (MCTS) Self-Refine iterative Self-Refine SR-MCTS pairwise rewarding model self-critic rewriting capabilities Pairwise Preference Reward Model (PPRM) Reinforcement Learning from Human Feedback (RLHF) Enhanced Borda Count (EBC) Global ranking Score ToT rStar GPQA AIME24 AMC23