title: Offline Reinforcement Learning for LLM Multi-Step Reasoning
summary: Improving the multi-step reasoning ability of large language models (LLMs)
  with offline reinforcement learning (RL) is essential for quickly adapting them
  to complex tasks. While Direct Preference Optimization (DPO) has shown promise
  in aligning LLMs with human preferences, it is less suitable for multi-step
  reasoning tasks because (1) DPO relies on paired preference data, which is not
  readily available for multi-step reasoning tasks, and (2) it treats all tokens
  uniformly, making it ineffective for credit assignment in multi-step reasoning
  tasks, which often come with sparse reward. In this work, we propose OREO
  (Offline Reasoning Optimization), an offline RL method for enhancing LLM
  multi-step reasoning. Building on insights from previous works of maximum
  entropy reinforcement learning, it jointly learns a policy model and value
  function by optimizing the soft Bellman Equation. We show in principle that it
  reduces the need to collect pairwise data and enables better credit assignment.
  Empirically, OREO surpasses existing offline learning methods on multi-step
  reasoning benchmarks, including mathematical reasoning tasks (GSM8K, MATH) and
  embodied agent control (ALFWorld). The approach can be extended to a
  multi-iteration framework when additional resources are available. Furthermore,
  the learned value function can be leveraged to guide the tree search for free,
  which can further boost performance during test time.
paper_authors._id: 6768f050bf7c0f8d9a17c4f2
paper_authors.name: Huaijie Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-23T11:31:56.207Z
paper_authors._id: 6310664063b70252b4779150
paper_authors.avatarUrl: /avatars/d514270c57b6ac494e0a419d792a72e5.svg
paper_authors.fullname: Huaijie Wang
paper_authors.type: user
paper_authors.user: jwhj
paper_authors._id: 6768f050bf7c0f8d9a17c4f3
paper_authors.name: Shibo Hao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-23T11:32:02.176Z
paper_authors._id: 660ee5df35d092e3fc2a3685
paper_authors.avatarUrl: /avatars/a7e0472fb7ea49973f74e3eea13dc964.svg
paper_authors.fullname: Shibo Hao
paper_authors.type: user
paper_authors.user: Shibo-UCSD
paper_authors._id: 6768f050bf7c0f8d9a17c4f4
paper_authors.name: Hanze Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-23T11:32:07.779Z
paper_authors._id: 63a3ff69f91ad3ea5703841d
paper_authors.avatarUrl: /avatars/69227c4bce01d33747c1377b6f9672db.svg
paper_authors.fullname: Hanze Dong
paper_authors.type: user
paper_authors.user: hendrydong
paper_authors._id: 6768f050bf7c0f8d9a17c4f5
paper_authors.name: Shenao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-23T11:09:50.829Z
paper_authors._id: 661213f894e0b3bff3e80c69
paper_authors.avatarUrl: /avatars/d8febbb081825bf91e487aa8bad3a391.svg
paper_authors.fullname: Shenao Zhang
paper_authors.type: user
paper_authors.user: ZhangShenao
paper_authors._id: 6768f050bf7c0f8d9a17c4f6
paper_authors.name: Yilin Bao
paper_authors._id: 6768f050bf7c0f8d9a17c4f7
paper_authors.name: Ziran Yang
paper_authors._id: 6768f050bf7c0f8d9a17c4f8
paper_authors.name: Yi Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-23T11:32:27.882Z
paper_authors._id: 62c88b04ab9c23f5c459ed90
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62c88b04ab9c23f5c459ed90/tEaeuKpXdXwqK-zq1H-8a.png
paper_authors.fullname: Yi Wu
paper_authors.type: user
paper_authors.user: yiwu
paper_title: Offline Reinforcement Learning for LLM Multi-Step Reasoning
paper_summary: Improving the multi-step reasoning ability of large language models (LLMs)
  with offline reinforcement learning (RL) is essential for quickly adapting them
  to complex tasks. While Direct Preference Optimization (DPO) has shown promise
  in aligning LLMs with human preferences, it is less suitable for multi-step
  reasoning tasks because (1) DPO relies on paired preference data, which is not
  readily available for multi-step reasoning tasks, and (2) it treats all tokens
  uniformly, making it ineffective for credit assignment in multi-step reasoning
  tasks, which often come with sparse reward. In this work, we propose OREO
  (Offline Reasoning Optimization), an offline RL method for enhancing LLM
  multi-step reasoning. Building on insights from previous works of maximum
  entropy reinforcement learning, it jointly learns a policy model and value
  function by optimizing the soft Bellman Equation. We show in principle that it
  reduces the need to collect pairwise data and enables better credit assignment.
  Empirically, OREO surpasses existing offline learning methods on multi-step
  reasoning benchmarks, including mathematical reasoning tasks (GSM8K, MATH) and
  embodied agent control (ALFWorld). The approach can be extended to a
  multi-iteration framework when additional resources are available. Furthermore,
  the learned value function can be leveraged to guide the tree search for free,
  which can further boost performance during test time.
paper_ai_keywords: Offline Reinforcement Learning (RL) Direct Preference Optimization (DPO) multi-step reasoning paired preference data credit assignment OREO (Offline Reasoning Optimization) maximum entropy reinforcement learning policy model value function soft Bellman Equation multi-step reasoning benchmarks mathematical reasoning tasks (GSM8K, MATH) embodied agent control (ALFWorld) multi-iteration framework tree search