title: Advancing LLM Reasoning Generalists with Preference Trees
summary: We introduce Eurus, a suite of large language models (LLMs) optimized for
  reasoning. Finetuned from Mistral-7B and CodeLlama-70B, Eurus models achieve
  state-of-the-art results among open-source models on a diverse set of
  benchmarks covering mathematics, code generation, and logical reasoning
  problems. Notably, Eurus-70B beats GPT-3.5 Turbo in reasoning through a
  comprehensive benchmarking across 12 tests covering five tasks, and achieves a
  33.3% pass@1 accuracy on LeetCode and 32.6% on TheoremQA, two challenging
  benchmarks, substantially outperforming existing open-source models by margins
  more than 13.3%. The strong performance of Eurus can be primarily attributed to
  UltraInteract, our newly-curated large-scale, high-quality alignment dataset
  specifically designed for complex reasoning tasks. UltraInteract can be used in
  both supervised fine-tuning and preference learning. For each instruction, it
  includes a preference tree consisting of (1) reasoning chains with diverse
  planning strategies in a unified format, (2) multi-turn interaction
  trajectories with the environment and the critique, and (3) pairwise data to
  facilitate preference learning. UltraInteract allows us to conduct an in-depth
  exploration of preference learning for reasoning tasks. Our investigation
  reveals that some well-established preference learning algorithms may be less
  suitable for reasoning tasks compared to their effectiveness in general
  conversations. Inspired by this, we derive a novel reward modeling objective
  which, together with UltraInteract, leads to a strong reward model.
paper_authors._id: 660cbd7d6601973e7a2a7a53
paper_authors.name: Lifan Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-04T09:47:43.785Z
paper_authors._id: 61506ce0b0e47650b93f79d0
paper_authors.avatarUrl: /avatars/21301cd9929a1a64f3f6427413855ea8.svg
paper_authors.fullname: Lievan
paper_authors.type: user
paper_authors.user: lievan
paper_authors._id: 660cbd7d6601973e7a2a7a54
paper_authors.name: Ganqu Cui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-24T11:12:19.514Z
paper_authors._id: 650eba9555dc1e841746f132
paper_authors.avatarUrl: /avatars/af6f5ee78f161d25ec0afc45d2def8eb.svg
paper_authors.fullname: Ganqu Cui
paper_authors.type: user
paper_authors.user: ganqu
paper_authors._id: 660cbd7d6601973e7a2a7a55
paper_authors.name: Hanbin Wang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-02-04T05:04:46.617Z
paper_authors._id: 6321152b8c0da827c72c7c16
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678783813705-6321152b8c0da827c72c7c16.jpeg
paper_authors.fullname: Hanbin Wang
paper_authors.type: user
paper_authors.user: hanbin
paper_authors._id: 660cbd7d6601973e7a2a7a56
paper_authors.name: Ning Ding
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-31T09:50:56.584Z
paper_authors._id: 60cf4bcb1ce3775ebb86e5d5
paper_authors.avatarUrl: /avatars/12bcd18d215abf91f297f93007733148.svg
paper_authors.fullname: Ning Ding
paper_authors.type: user
paper_authors.user: stingning
paper_authors._id: 660cbd7d6601973e7a2a7a57
paper_authors.name: Xingyao Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-25T07:39:14.796Z
paper_authors._id: 62c63f31031996c36c848504
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62c63f31031996c36c848504/xZvH6jnozuFLXbtRSU-h1.jpeg
paper_authors.fullname: Xingyao Wang
paper_authors.type: user
paper_authors.user: xingyaoww
paper_authors._id: 660cbd7d6601973e7a2a7a58
paper_authors.name: Jia Deng
paper_authors._id: 660cbd7d6601973e7a2a7a59
paper_authors.name: Boji Shan
paper_authors._id: 660cbd7d6601973e7a2a7a5a
paper_authors.name: Huimin Chen
paper_authors._id: 660cbd7d6601973e7a2a7a5b
paper_authors.name: Ruobing Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-05T09:02:06.930Z
paper_authors._id: 6622443b9b0614a760dd8123
paper_authors.avatarUrl: /avatars/acb6c1c9c429af1112530dcf76a8e420.svg
paper_authors.fullname: Ruobing Xie
paper_authors.type: user
paper_authors.user: Ruobing-Xie
paper_authors._id: 660cbd7d6601973e7a2a7a5c
paper_authors.name: Yankai Lin
paper_authors._id: 660cbd7d6601973e7a2a7a5d
paper_authors.name: Zhenghao Liu
paper_authors._id: 660cbd7d6601973e7a2a7a5e
paper_authors.name: Bowen Zhou
paper_authors._id: 660cbd7d6601973e7a2a7a5f
paper_authors.name: Hao Peng
paper_authors._id: 660cbd7d6601973e7a2a7a60
paper_authors.name: Zhiyuan Liu
paper_authors._id: 660cbd7d6601973e7a2a7a61
paper_authors.name: Maosong Sun
paper_title: Advancing LLM Reasoning Generalists with Preference Trees
paper_summary: We introduce Eurus, a suite of large language models (LLMs) optimized for
  reasoning. Finetuned from Mistral-7B and CodeLlama-70B, Eurus models achieve
  state-of-the-art results among open-source models on a diverse set of
  benchmarks covering mathematics, code generation, and logical reasoning
  problems. Notably, Eurus-70B beats GPT-3.5 Turbo in reasoning through a
  comprehensive benchmarking across 12 tests covering five tasks, and achieves a
  33.3% pass@1 accuracy on LeetCode and 32.6% on TheoremQA, two challenging
  benchmarks, substantially outperforming existing open-source models by margins
  more than 13.3%. The strong performance of Eurus can be primarily attributed to
  UltraInteract, our newly-curated large-scale, high-quality alignment dataset
  specifically designed for complex reasoning tasks. UltraInteract can be used in
  both supervised fine-tuning and preference learning. For each instruction, it
  includes a preference tree consisting of (1) reasoning chains with diverse
  planning strategies in a unified format, (2) multi-turn interaction
  trajectories with the environment and the critique, and (3) pairwise data to
  facilitate preference learning. UltraInteract allows us to conduct an in-depth
  exploration of preference learning for reasoning tasks. Our investigation
  reveals that some well-established preference learning algorithms may be less
  suitable for reasoning tasks compared to their effectiveness in general
  conversations. Inspired by this, we derive a novel reward modeling objective
  which, together with UltraInteract, leads to a strong reward model.
paper_ai_keywords: large language models (LLMs) Mistral-7B CodeLlama-70B comprehensive benchmarking reasoning LeetCode TheoremQA UltraInteract large-scale high-quality alignment dataset complex reasoning tasks supervised fine-tuning preference learning reasoning chains multi-turn interaction trajectories pairwise data preference tree novel reward modeling objective reward model