title: Building Math Agents with Multi-Turn Iterative Preference Learning
summary: Recent studies have shown that large language models' (LLMs) mathematical
  problem-solving capabilities can be enhanced by integrating external tools,
  such as code interpreters, and employing multi-turn Chain-of-Thought (CoT)
  reasoning. While current methods focus on synthetic data generation and
  Supervised Fine-Tuning (SFT), this paper studies the complementary direct
  preference learning approach to further improve model performance. However,
  existing direct preference learning algorithms are originally designed for the
  single-turn chat task, and do not fully address the complexities of multi-turn
  reasoning and external tool integration required for tool-integrated
  mathematical reasoning tasks. To fill in this gap, we introduce a multi-turn
  direct preference learning framework, tailored for this context, that leverages
  feedback from code interpreters and optimizes trajectory-level preferences.
  This framework includes multi-turn DPO and multi-turn KTO as specific
  implementations. The effectiveness of our framework is validated through
  training of various language models using an augmented prompt set from the
  GSM8K and MATH datasets. Our results demonstrate substantial improvements: a
  supervised fine-tuned Gemma-1.1-it-7B model's performance increased from 77.5%
  to 83.9% on GSM8K and from 46.1% to 51.2% on MATH. Similarly, a Gemma-2-it-9B
  model improved from 84.1% to 86.3% on GSM8K and from 51.0% to 54.5% on MATH.
paper_authors._id: 66db1301c39b38d37f79b11b
paper_authors.name: Wei Xiong
paper_authors._id: 66db1301c39b38d37f79b11c
paper_authors.name: Chengshuai Shi
paper_authors._id: 66db1301c39b38d37f79b11d
paper_authors.name: Jiaming Shen
paper_authors._id: 66db1301c39b38d37f79b11e
paper_authors.name: Aviv Rosenberg
paper_authors._id: 66db1301c39b38d37f79b11f
paper_authors.name: Zhen Qin
paper_authors._id: 66db1301c39b38d37f79b120
paper_authors.name: Daniele Calandriello
paper_authors._id: 66db1301c39b38d37f79b121
paper_authors.name: Misha Khalman
paper_authors._id: 66db1301c39b38d37f79b122
paper_authors.name: Rishabh Joshi
paper_authors._id: 66db1301c39b38d37f79b123
paper_authors.name: Bilal Piot
paper_authors._id: 66db1301c39b38d37f79b124
paper_authors.name: Mohammad Saleh
paper_authors._id: 66db1301c39b38d37f79b125
paper_authors.name: Chi Jin
paper_authors._id: 66db1301c39b38d37f79b126
paper_authors.name: Tong Zhang
paper_authors._id: 66db1301c39b38d37f79b127
paper_authors.name: Tianqi Liu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-09-06T14:45:51.022Z
paper_authors._id: 646784af696e7355f5d5001f
paper_authors.avatarUrl: /avatars/39ce8999c296584ee0bdeb7848eee6d9.svg
paper_authors.fullname: Tianqi Liu
paper_authors.type: user
paper_authors.user: TianqiLiuAI
paper_title: Building Math Agents with Multi-Turn Iterative Preference Learning
paper_summary: Recent studies have shown that large language models' (LLMs) mathematical
  problem-solving capabilities can be enhanced by integrating external tools,
  such as code interpreters, and employing multi-turn Chain-of-Thought (CoT)
  reasoning. While current methods focus on synthetic data generation and
  Supervised Fine-Tuning (SFT), this paper studies the complementary direct
  preference learning approach to further improve model performance. However,
  existing direct preference learning algorithms are originally designed for the
  single-turn chat task, and do not fully address the complexities of multi-turn
  reasoning and external tool integration required for tool-integrated
  mathematical reasoning tasks. To fill in this gap, we introduce a multi-turn
  direct preference learning framework, tailored for this context, that leverages
  feedback from code interpreters and optimizes trajectory-level preferences.
  This framework includes multi-turn DPO and multi-turn KTO as specific
  implementations. The effectiveness of our framework is validated through
  training of various language models using an augmented prompt set from the
  GSM8K and MATH datasets. Our results demonstrate substantial improvements: a
  supervised fine-tuned Gemma-1.1-it-7B model's performance increased from 77.5%
  to 83.9% on GSM8K and from 46.1% to 51.2% on MATH. Similarly, a Gemma-2-it-9B
  model improved from 84.1% to 86.3% on GSM8K and from 51.0% to 54.5% on MATH.
paper_ai_summary: A multi-turn direct preference learning framework, incorporating feedback from code interpreters, enhances large language models' mathematical problem-solving capabilities with substantial performance improvements.
paper_ai_keywords: large language models Chain-of-Thought reasoning Supervised Fine-Tuning direct preference learning multi-turn DPO KTO trajectory-level preferences GSM8K MATH Gemma-1.1-it-7B Gemma-2-it-9B