title: Fine-Grained Human Feedback Gives Better Rewards for Language Model
    Training
summary: Language models (LMs) often exhibit undesirable text generation behaviors,
  including generating false, toxic, or irrelevant outputs. Reinforcement
  learning from human feedback (RLHF) - where human preference judgments on LM
  outputs are transformed into a learning signal - has recently shown promise in
  addressing these issues. However, such holistic feedback conveys limited
  information on long text outputs; it does not indicate which aspects of the
  outputs influenced user preference; e.g., which parts contain what type(s) of
  errors. In this paper, we use fine-grained human feedback (e.g., which sentence
  is false, which sub-sentence is irrelevant) as an explicit training signal. We
  introduce Fine-Grained RLHF, a framework that enables training and learning
  from reward functions that are fine-grained in two respects: (1) density,
  providing a reward after every segment (e.g., a sentence) is generated; and (2)
  incorporating multiple reward models associated with different feedback types
  (e.g., factual incorrectness, irrelevance, and information incompleteness). We
  conduct experiments on detoxification and long-form question answering to
  illustrate how learning with such reward functions leads to improved
  performance, supported by both automatic and human evaluation. Additionally, we
  show that LM behaviors can be customized using different combinations of
  fine-grained reward models. We release all data, collected human feedback, and
  codes at https://FineGrainedRLHF.github.io.
paper_authors._id: 647d58da36e109abce40dc99
paper_authors.name: Zeqiu Wu
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-06-05T03:39:07.589Z
paper_authors._id: 647d335836e109abce3ffb79
paper_authors.avatarUrl: /avatars/fee73f1e15502ca51f8a7a928f653d31.svg
paper_authors.fullname: Zeqiu Wu
paper_authors.type: user
paper_authors.user: zeqiuwu1
paper_authors._id: 647d58da36e109abce40dc9a
paper_authors.name: Yushi Hu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-06-05T09:56:38.574Z
paper_authors._id: 62b1474bdcbad6848a91a54e
paper_authors.avatarUrl: /avatars/d7308899b46232cad4a48a0e876449a8.svg
paper_authors.fullname: Yushi Hu
paper_authors.type: user
paper_authors.user: yushihu
paper_authors._id: 647d58da36e109abce40dc9b
paper_authors.name: Weijia Shi
paper_authors._id: 647d58da36e109abce40dc9c
paper_authors.name: Nouha Dziri
paper_authors._id: 647d58da36e109abce40dc9d
paper_authors.name: Alane Suhr
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-09T20:08:55.528Z
paper_authors._id: 6611e6e1188ff298b0dd0b79
paper_authors.avatarUrl: /avatars/3a495283955ec9e06e1829c7eb2cd9a4.svg
paper_authors.fullname: Alane Suhr
paper_authors.type: user
paper_authors.user: alsuhr
paper_authors._id: 647d58da36e109abce40dc9e
paper_authors.name: Prithviraj Ammanabrolu
paper_authors._id: 647d58da36e109abce40dc9f
paper_authors.name: Noah A. Smith
paper_authors._id: 647d58da36e109abce40dca0
paper_authors.name: Mari Ostendorf
paper_authors._id: 647d58da36e109abce40dca1
paper_authors.name: Hannaneh Hajishirzi
paper_title: Fine-Grained Human Feedback Gives Better Rewards for Language Model
    Training
paper_summary: Language models (LMs) often exhibit undesirable text generation behaviors,
  including generating false, toxic, or irrelevant outputs. Reinforcement
  learning from human feedback (RLHF) - where human preference judgments on LM
  outputs are transformed into a learning signal - has recently shown promise in
  addressing these issues. However, such holistic feedback conveys limited
  information on long text outputs; it does not indicate which aspects of the
  outputs influenced user preference; e.g., which parts contain what type(s) of
  errors. In this paper, we use fine-grained human feedback (e.g., which sentence
  is false, which sub-sentence is irrelevant) as an explicit training signal. We
  introduce Fine-Grained RLHF, a framework that enables training and learning
  from reward functions that are fine-grained in two respects: (1) density,
  providing a reward after every segment (e.g., a sentence) is generated; and (2)
  incorporating multiple reward models associated with different feedback types
  (e.g., factual incorrectness, irrelevance, and information incompleteness). We
  conduct experiments on detoxification and long-form question answering to
  illustrate how learning with such reward functions leads to improved
  performance, supported by both automatic and human evaluation. Additionally, we
  show that LM behaviors can be customized using different combinations of
  fine-grained reward models. We release all data, collected human feedback, and
  codes at https://FineGrainedRLHF.github.io.
paper_ai_keywords: Reinforcement learning from human feedback (RLHF) fine-grained human feedback Fine-Grained RLHF reward functions density sentence segments multiple reward models factual incorrectness irrelevance information incompleteness detoxification long-form question answering automatic evaluation human evaluation customized LM behaviors