Learning Explainable Dense Reward Shapes via Bayesian Optimization
Current reinforcement learning from human feedback (RLHF) pipelines for large
language model (LLM) alignment typically assign scalar rewards to sequences,
using the final token as a surrogate indicator for the quality of the entire
sequence. However, this leads to sparse feedback and suboptimal token-level
credit assignment. In this work, we frame reward shaping as an optimization
problem focused on token-level credit assignment. We propose a reward-shaping
function leveraging explainability methods such as SHAP and LIME to estimate
per-token rewards from the reward model. To learn parameters of this shaping
function, we employ a bilevel optimization framework that integrates Bayesian
Optimization and policy training to handle noise from the token reward
estimates. Our experiments show that achieving a better balance of token-level
reward attribution leads to performance improvements over baselines on
downstream tasks and finds an optimal policy faster during training.
Furthermore, we show theoretically that explainability methods that are feature
additive attribution functions maintain the optimal policy as the original
reward.
68127237228929e037039e84
Ryan Koo
claimed_verified
2025-05-02T06:36:01.162Z
63ed75f3679c2cc40ab9e40f
/avatars/e70bc88e72082e5ad5a106bc92a27ab1.svg
Ryan Koo
user
rngusry
68127237228929e037039e85
Ian Yang
extracted_pending
2025-04-30T18:55:52.538Z
649339184005b1c3c021529a
/avatars/67391550642d2a476d388b4898134b8c.svg
Ian Yang
user
iyang30
68127237228929e037039e86
Vipul Raheja
68127237228929e037039e87
Mingyi Hong
claimed_verified
2025-06-02T07:50:27.282Z
679a5fa807076ced8ae5f41b
/avatars/63cac0a05c9c7500c9ab9c75667a38d7.svg
Mingyi Hong
user
poborskypaul
68127237228929e037039e88
Kwang-Sung Jun
68127237228929e037039e89
Dongyeop Kang
Learning Explainable Dense Reward Shapes via Bayesian Optimization
Current reinforcement learning from human feedback (RLHF) pipelines for large
language model (LLM) alignment typically assign scalar rewards to sequences,
using the final token as a surrogate indicator for the quality of the entire
sequence. However, this leads to sparse feedback and suboptimal token-level
credit assignment. In this work, we frame reward shaping as an optimization
problem focused on token-level credit assignment. We propose a reward-shaping
function leveraging explainability methods such as SHAP and LIME to estimate
per-token rewards from the reward model. To learn parameters of this shaping
function, we employ a bilevel optimization framework that integrates Bayesian
Optimization and policy training to handle noise from the token reward
estimates. Our experiments show that achieving a better balance of token-level
reward attribution leads to performance improvements over baselines on
downstream tasks and finds an optimal policy faster during training.
Furthermore, we show theoretically that explainability methods that are feature
additive attribution functions maintain the optimal policy as the original
reward.
Proposing a reward-shaping function using explainability methods to improve token-level credit assignment in reinforcement learning from human feedback for better performance and faster policy learning.
reinforcement learning from human feedback large language model reward shaping token-level credit assignment SHAP LIME bilevel optimization Bayesian Optimization feature additive attribution functions optimal policy