title: Segmenting Text and Learning Their Rewards for Improved RLHF in Language
    Model
summary: Reinforcement learning from human feedback (RLHF) has been widely adopted to
  align language models (LMs) with human preference. Prior RLHF works typically
  take a bandit formulation, which, though intuitive, ignores the sequential
  nature of LM generation and can suffer from the sparse reward issue. While
  recent works propose dense token-level RLHF, treating each token as an action
  may be oversubtle to proper reward assignment. In this paper, we seek to get
  the best of both by training and utilizing a segment-level reward model, which
  assigns a reward to each semantically complete text segment that spans over a
  short sequence of tokens. For reward learning, our method allows dynamic text
  segmentation and compatibility with standard sequence-preference datasets. For
  effective RL-based LM training against segment reward, we generalize the
  classical scalar bandit reward normalizers into location-aware normalizer
  functions and interpolate the segment reward for further densification. With
  these designs, our method performs competitively on three popular RLHF
  benchmarks for LM policy: AlpacaEval 2.0, Arena-Hard, and MT-Bench. Ablation
  studies are conducted to further demonstrate our method.
paper_authors._id: 677dea23e86d0754dc6e3f09
paper_authors.name: Yueqin Yin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-08T09:44:41.339Z
paper_authors._id: 605e8dfd5abeb13e714c4c18
paper_authors.avatarUrl: /avatars/bc27a0ed17b2bd4311e89d3028fa327b.svg
paper_authors.fullname: yueqin yin
paper_authors.type: user
paper_authors.user: yyqoni
paper_authors._id: 677dea23e86d0754dc6e3f0a
paper_authors.name: Shentao Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-08T21:08:25.725Z
paper_authors._id: 677ebaf3fcaae73ddd6c8475
paper_authors.avatarUrl: /avatars/98db20fe3bbbf7caffb5821a7242dc54.svg
paper_authors.fullname: Shentao Yang
paper_authors.type: user
paper_authors.user: shentaoyang
paper_authors._id: 677dea23e86d0754dc6e3f0b
paper_authors.name: Yujia Xie
paper_authors._id: 677dea23e86d0754dc6e3f0c
paper_authors.name: Ziyi Yang
paper_authors._id: 677dea23e86d0754dc6e3f0d
paper_authors.name: Yuting Sun
paper_authors._id: 677dea23e86d0754dc6e3f0e
paper_authors.name: Hany Awadalla
paper_authors._id: 677dea23e86d0754dc6e3f0f
paper_authors.name: Weizhu Chen
paper_authors._id: 677dea23e86d0754dc6e3f10
paper_authors.name: Mingyuan Zhou
paper_title: Segmenting Text and Learning Their Rewards for Improved RLHF in Language
    Model
paper_summary: Reinforcement learning from human feedback (RLHF) has been widely adopted to
  align language models (LMs) with human preference. Prior RLHF works typically
  take a bandit formulation, which, though intuitive, ignores the sequential
  nature of LM generation and can suffer from the sparse reward issue. While
  recent works propose dense token-level RLHF, treating each token as an action
  may be oversubtle to proper reward assignment. In this paper, we seek to get
  the best of both by training and utilizing a segment-level reward model, which
  assigns a reward to each semantically complete text segment that spans over a
  short sequence of tokens. For reward learning, our method allows dynamic text
  segmentation and compatibility with standard sequence-preference datasets. For
  effective RL-based LM training against segment reward, we generalize the
  classical scalar bandit reward normalizers into location-aware normalizer
  functions and interpolate the segment reward for further densification. With
  these designs, our method performs competitively on three popular RLHF
  benchmarks for LM policy: AlpacaEval 2.0, Arena-Hard, and MT-Bench. Ablation
  studies are conducted to further demonstrate our method.
paper_ai_keywords: Reinforcement learning from human feedback (RLHF) language models (LMs) bandit formulation sparse reward issue dense token-level RLHF segment-level reward model semantically complete text segment dynamic text segmentation standard sequence-preference datasets scalar bandit reward normalizers location-aware normalizer functions segment reward reward learning RLHF benchmarks AlpacaEval 2.0 Arena-Hard MT-Bench