title: Learning Explainable Dense Reward Shapes via Bayesian Optimization
summary: Current reinforcement learning from human feedback (RLHF) pipelines for large
  language model (LLM) alignment typically assign scalar rewards to sequences,
  using the final token as a surrogate indicator for the quality of the entire
  sequence. However, this leads to sparse feedback and suboptimal token-level
  credit assignment. In this work, we frame reward shaping as an optimization
  problem focused on token-level credit assignment. We propose a reward-shaping
  function leveraging explainability methods such as SHAP and LIME to estimate
  per-token rewards from the reward model. To learn parameters of this shaping
  function, we employ a bilevel optimization framework that integrates Bayesian
  Optimization and policy training to handle noise from the token reward
  estimates. Our experiments show that achieving a better balance of token-level
  reward attribution leads to performance improvements over baselines on
  downstream tasks and finds an optimal policy faster during training.
  Furthermore, we show theoretically that explainability methods that are feature
  additive attribution functions maintain the optimal policy as the original
  reward.
paper_authors._id: 68127237228929e037039e84
paper_authors.name: Ryan Koo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-02T06:36:01.162Z
paper_authors._id: 63ed75f3679c2cc40ab9e40f
paper_authors.avatarUrl: /avatars/e70bc88e72082e5ad5a106bc92a27ab1.svg
paper_authors.fullname: Ryan Koo
paper_authors.type: user
paper_authors.user: rngusry
paper_authors._id: 68127237228929e037039e85
paper_authors.name: Ian Yang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-04-30T18:55:52.538Z
paper_authors._id: 649339184005b1c3c021529a
paper_authors.avatarUrl: /avatars/67391550642d2a476d388b4898134b8c.svg
paper_authors.fullname: Ian Yang
paper_authors.type: user
paper_authors.user: iyang30
paper_authors._id: 68127237228929e037039e86
paper_authors.name: Vipul Raheja
paper_authors._id: 68127237228929e037039e87
paper_authors.name: Mingyi Hong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:50:27.282Z
paper_authors._id: 679a5fa807076ced8ae5f41b
paper_authors.avatarUrl: /avatars/63cac0a05c9c7500c9ab9c75667a38d7.svg
paper_authors.fullname: Mingyi Hong
paper_authors.type: user
paper_authors.user: poborskypaul
paper_authors._id: 68127237228929e037039e88
paper_authors.name: Kwang-Sung Jun
paper_authors._id: 68127237228929e037039e89
paper_authors.name: Dongyeop Kang
paper_title: Learning Explainable Dense Reward Shapes via Bayesian Optimization
paper_summary: Current reinforcement learning from human feedback (RLHF) pipelines for large
  language model (LLM) alignment typically assign scalar rewards to sequences,
  using the final token as a surrogate indicator for the quality of the entire
  sequence. However, this leads to sparse feedback and suboptimal token-level
  credit assignment. In this work, we frame reward shaping as an optimization
  problem focused on token-level credit assignment. We propose a reward-shaping
  function leveraging explainability methods such as SHAP and LIME to estimate
  per-token rewards from the reward model. To learn parameters of this shaping
  function, we employ a bilevel optimization framework that integrates Bayesian
  Optimization and policy training to handle noise from the token reward
  estimates. Our experiments show that achieving a better balance of token-level
  reward attribution leads to performance improvements over baselines on
  downstream tasks and finds an optimal policy faster during training.
  Furthermore, we show theoretically that explainability methods that are feature
  additive attribution functions maintain the optimal policy as the original
  reward.
paper_ai_summary: Proposing a reward-shaping function using explainability methods to improve token-level credit assignment in reinforcement learning from human feedback for better performance and faster policy learning.
paper_ai_keywords: reinforcement learning from human feedback large language model reward shaping token-level credit assignment SHAP LIME bilevel optimization Bayesian Optimization feature additive attribution functions optimal policy