title: Iterative Value Function Optimization for Guided Decoding
summary: While Reinforcement Learning from Human Feedback (RLHF) has become the
  predominant method for controlling language model outputs, it suffers from high
  computational costs and training instability. Guided decoding, especially
  value-guided methods, offers a cost-effective alternative by controlling
  outputs without re-training models. However, the accuracy of the value function
  is crucial for value-guided decoding, as inaccuracies can lead to suboptimal
  decision-making and degraded performance. Existing methods struggle with
  accurately estimating the optimal value function, leading to less effective
  control. We propose Iterative Value Function Optimization, a novel framework
  that addresses these limitations through two key components: Monte Carlo Value
  Estimation, which reduces estimation variance by exploring diverse
  trajectories, and Iterative On-Policy Optimization, which progressively
  improves value estimation through collecting trajectories from value-guided
  policies. Extensive experiments on text summarization, multi-turn dialogue, and
  instruction following demonstrate the effectiveness of value-guided decoding
  approaches in aligning language models. These approaches not only achieve
  alignment but also significantly reduce computational costs by leveraging
  principled value function optimization for efficient and effective control.
paper_authors._id: 67c80f94ccc2e04adfa67079
paper_authors.name: Zhenhua Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T10:42:35.402Z
paper_authors._id: 619b00dd4b0db5ca9d3ea35f
paper_authors.avatarUrl: /avatars/fce5ac388b7f10cbbc63e9992a5a799f.svg
paper_authors.fullname: Zhenhua Liu
paper_authors.type: user
paper_authors.user: zhliu
paper_authors._id: 67c80f94ccc2e04adfa6707a
paper_authors.name: Lijun Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T10:41:31.679Z
paper_authors._id: 65ca1fbbecfd8608f5b89666
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/YM6yIAjVQCRHjk6uJd1jr.jpeg
paper_authors.fullname: lilijun
paper_authors.type: user
paper_authors.user: lljhbxt
paper_authors._id: 67c80f94ccc2e04adfa6707b
paper_authors.name: Ruizhe Chen
paper_authors._id: 67c80f94ccc2e04adfa6707c
paper_authors.name: Yuxian Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T10:40:51.943Z
paper_authors._id: 64a67bff9f3b568c203cb533
paper_authors.avatarUrl: /avatars/630d117a6e4392b78b35d6b95066b52a.svg
paper_authors.fullname: YU-XIAN JIANG
paper_authors.type: user
paper_authors.user: ab3223323
paper_authors._id: 67c80f94ccc2e04adfa6707d
paper_authors.name: Tong Zhu
paper_authors._id: 67c80f94ccc2e04adfa6707e
paper_authors.name: Wenliang Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T10:40:45.604Z
paper_authors._id: 63477237511cd17d2c7a1be9
paper_authors.avatarUrl: /avatars/97cb3f640ded58344bca0ab279473134.svg
paper_authors.fullname: Wenliang Chen
paper_authors.type: user
paper_authors.user: jokephp
paper_authors._id: 67c80f94ccc2e04adfa6707f
paper_authors.name: Jing Shao
paper_title: Iterative Value Function Optimization for Guided Decoding
paper_summary: While Reinforcement Learning from Human Feedback (RLHF) has become the
  predominant method for controlling language model outputs, it suffers from high
  computational costs and training instability. Guided decoding, especially
  value-guided methods, offers a cost-effective alternative by controlling
  outputs without re-training models. However, the accuracy of the value function
  is crucial for value-guided decoding, as inaccuracies can lead to suboptimal
  decision-making and degraded performance. Existing methods struggle with
  accurately estimating the optimal value function, leading to less effective
  control. We propose Iterative Value Function Optimization, a novel framework
  that addresses these limitations through two key components: Monte Carlo Value
  Estimation, which reduces estimation variance by exploring diverse
  trajectories, and Iterative On-Policy Optimization, which progressively
  improves value estimation through collecting trajectories from value-guided
  policies. Extensive experiments on text summarization, multi-turn dialogue, and
  instruction following demonstrate the effectiveness of value-guided decoding
  approaches in aligning language models. These approaches not only achieve
  alignment but also significantly reduce computational costs by leveraging
  principled value function optimization for efficient and effective control.
paper_ai_summary: A novel framework, Iterative Value Function Optimization, improves value-guided decoding in reinforcement learning by enhancing value function estimation accuracy, aligning language models and reducing computational costs.
paper_ai_keywords: Reinforcement Learning from Human Feedback (RLHF) value-guided methods value function Monte Carlo Value Estimation Iterative On-Policy Optimization text summarization multi-turn dialogue instruction following