title: Lean and Mean: Decoupled Value Policy Optimization with Global Value
    Guidance
summary: Proximal Policy Optimization (PPO)-based Reinforcement Learning from Human
  Feedback (RLHF) is essential for aligning large language models (LLMs) with
  human preferences. It requires joint training of an actor and critic with a
  pretrained, fixed reward model for guidance. This approach increases
  computational complexity and instability due to actor-critic interdependence.
  Additionally, PPO lacks access to true environment rewards in LLM tasks,
  limiting its adaptability. Under such conditions, pretraining a value model or
  a reward model becomes equivalent, as both provide fixed supervisory signals
  without new ground-truth feedback. To address these issues, we propose
  Decoupled Value Policy Optimization (DVPO), a lean framework that
  replaces traditional reward modeling with a pretrained global value model
  (GVM). The GVM is conditioned on policy trajectories and predicts token-level
  return-to-go estimates. By decoupling value model from policy training (via
  frozen GVM-driven RL objectives), DVPO eliminates actor-critic interdependence,
  reducing GPU memory usage by 40\% and training time by 35\% compared to
  conventional RLHF. Experiments across benchmarks show DVPO outperforms
  efficient RLHF methods (e.g., DPO) while matching state-of-the-art PPO in
  performance.
paper_authors._id: 67be807e8a5a805423137ca2
paper_authors.name: Chenghua Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:30:31.972Z
paper_authors._id: 664af07a691370727c281031
paper_authors.avatarUrl: /avatars/e5ed17342e0ea953bacc7d57e9f3b686.svg
paper_authors.fullname: Cheng Hua Huang
paper_authors.type: user
paper_authors.user: LanceZomax
paper_authors._id: 67be807e8a5a805423137ca3
paper_authors.name: Lu Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:30:39.129Z
paper_authors._id: 6406afa8a577649430c64363
paper_authors.avatarUrl: /avatars/9bd1768c91d509c8c49970e9fd7775a5.svg
paper_authors.fullname: LuWang
paper_authors.type: user
paper_authors.user: LuWang
paper_authors._id: 67be807e8a5a805423137ca4
paper_authors.name: Fangkai Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-27T09:17:46.382Z
paper_authors._id: 669dcf6200970c3b27aafa5d
paper_authors.avatarUrl: /avatars/bb9ed5ff86326fdaeb184c6b0e40f74f.svg
paper_authors.fullname: kaikai yang
paper_authors.type: user
paper_authors.user: keanudicap
paper_authors._id: 67be807e8a5a805423137ca5
paper_authors.name: Pu Zhao
paper_authors._id: 67be807e8a5a805423137ca6
paper_authors.name: Zhixu Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:30:46.084Z
paper_authors._id: 661dd71d2ae9013218415e6f
paper_authors.avatarUrl: /avatars/389883e5886628c07cb0b08fc8c93c3b.svg
paper_authors.fullname: Zhixu Li
paper_authors.type: user
paper_authors.user: ZhixuLi
paper_authors._id: 67be807e8a5a805423137ca7
paper_authors.name: Qingwei Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:31:00.637Z
paper_authors._id: 652fc9f39bc50a6c0e435224
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/652fc9f39bc50a6c0e435224/70OBVDHHBsxG2giJ-E3_1.jpeg
paper_authors.fullname: Lin Qingwei
paper_authors.type: user
paper_authors.user: Eliblo1969
paper_authors._id: 67be807e8a5a805423137ca8
paper_authors.name: Dongmei Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:31:14.045Z
paper_authors._id: 66473d2c7abe6ad66e81a3dd
paper_authors.avatarUrl: /avatars/82f40244806c06ffeaa1c4265e9725ea.svg
paper_authors.fullname: ZHANGDONGMEI
paper_authors.type: user
paper_authors.user: ZDM6426
paper_authors._id: 67be807e8a5a805423137ca9
paper_authors.name: Saravan Rajmohan
paper_authors._id: 67be807e8a5a805423137caa
paper_authors.name: Qi Zhang
paper_title: Lean and Mean: Decoupled Value Policy Optimization with Global Value
    Guidance
paper_summary: Proximal Policy Optimization (PPO)-based Reinforcement Learning from Human
  Feedback (RLHF) is essential for aligning large language models (LLMs) with
  human preferences. It requires joint training of an actor and critic with a
  pretrained, fixed reward model for guidance. This approach increases
  computational complexity and instability due to actor-critic interdependence.
  Additionally, PPO lacks access to true environment rewards in LLM tasks,
  limiting its adaptability. Under such conditions, pretraining a value model or
  a reward model becomes equivalent, as both provide fixed supervisory signals
  without new ground-truth feedback. To address these issues, we propose
  Decoupled Value Policy Optimization (DVPO), a lean framework that
  replaces traditional reward modeling with a pretrained global value model
  (GVM). The GVM is conditioned on policy trajectories and predicts token-level
  return-to-go estimates. By decoupling value model from policy training (via
  frozen GVM-driven RL objectives), DVPO eliminates actor-critic interdependence,
  reducing GPU memory usage by 40\% and training time by 35\% compared to
  conventional RLHF. Experiments across benchmarks show DVPO outperforms
  efficient RLHF methods (e.g., DPO) while matching state-of-the-art PPO in
  performance.
paper_ai_keywords: Proximal Policy Optimization (PPO) Reinforcement Learning from Human Feedback (RLHF) large language models (LLMs) actor critic pretrained, fixed reward model sparse crucial token-level return-to-go Decoupled Value Policy Optimization (DVPO) global value model (GVM) frozen GVM-driven RL objectives actor-critic interdependence