LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoning
<|reference_start|>LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoning: This paper presents an advanced mathematical problem-solving framework, LLaMA-Berry, for enhancing the mathematical reasoning ability of Large Language Models (LLMs). The framework combines Monte Carlo Tree Search (MCTS) with iterative Self-Refine to optimize the reasoning path and utilizes a pairwise reward model to evaluate different paths globally. By leveraging the self-critic and rewriting capabilities of LLMs, Self-Refine applied to MCTS (SR-MCTS) overcomes the inefficiencies and limitations of conventional step-wise and greedy search algorithms by fostering a more efficient exploration of solution spaces. Pairwise Preference Reward Model~(PPRM), inspired by Reinforcement Learning from Human Feedback (RLHF), is then used to model pairwise preferences between solutions, utilizing an Enhanced Borda Count (EBC) method to synthesize these preferences into a global ranking score to find better answers. This approach addresses the challenges of scoring variability and non-independent distributions in mathematical reasoning tasks. The framework has been tested on general and advanced benchmarks, showing superior performance in terms of search efficiency and problem-solving capability compared to existing methods like ToT and rStar, particularly in complex Olympiad-level benchmarks, including GPQA, AIME24 and AMC23.<|reference_end|>
@article{zhang2024llama-berry:,
    title={LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level
  Mathematical Reasoning},
    author={Di Zhang, Jianbo Wu, Jingdi Lei, Tong Che, Jiatong Li, Tong Xie,
  Xiaoshui Huang, Shufei Zhang, Marco Pavone, Yuqiang Li, Wanli Ouyang,
  Dongzhan Zhou},
    journal={arXiv preprint arXiv:2410.02884},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.02884},
    primaryClass={cs.AI cs.CL}
}