title: Enable Language Models to Implicitly Learn Self-Improvement From Data
summary: Large Language Models (LLMs) have demonstrated remarkable capabilities in
  open-ended text generation tasks. However, the inherent open-ended nature of
  these tasks implies that there is always room for improvement in the quality of
  model responses. To address this challenge, various approaches have been
  proposed to enhance the performance of LLMs. There has been a growing focus on
  enabling LLMs to self-improve their response quality, thereby reducing the
  reliance on extensive human annotation efforts for collecting diverse and
  high-quality training data. Recently, prompting-based methods have been widely
  explored among self-improvement methods owing to their effectiveness,
  efficiency, and convenience. However, those methods usually require explicitly
  and thoroughly written rubrics as inputs to LLMs. It is expensive and
  challenging to manually derive and provide all necessary rubrics with a
  real-world complex goal for improvement (e.g., being more helpful and less
  harmful). To this end, we propose an ImPlicit Self-ImprovemenT (PIT) framework
  that implicitly learns the improvement goal from human preference data. PIT
  only requires preference data that are used to train reward models without
  extra human efforts. Specifically, we reformulate the training objective of
  reinforcement learning from human feedback (RLHF) -- instead of maximizing
  response quality for a given input, we maximize the quality gap of the response
  conditioned on a reference response. In this way, PIT is implicitly trained
  with the improvement goal of better aligning with human preferences.
  Experiments on two real-world datasets and one synthetic dataset show that our
  method significantly outperforms prompting-based methods.
paper_authors._id: 651b9345b9ae9f37afb52eef
paper_authors.name: Ziqi Wang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-10-03T10:11:23.704Z
paper_authors._id: 651c321c4e15c57e1b0efca6
paper_authors.avatarUrl: /avatars/241c4d975356bd986bad7303e1534bce.svg
paper_authors.fullname: Ziqi wang
paper_authors.type: user
paper_authors.user: wzq016
paper_authors._id: 651b9345b9ae9f37afb52ef0
paper_authors.name: Le Hou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-03T10:10:02.555Z
paper_authors._id: 626b2402c804c432c1b5004c
paper_authors.avatarUrl: /avatars/a119644422335e260af0ca4ef6399870.svg
paper_authors.fullname: Le Hou
paper_authors.type: user
paper_authors.user: solooobear
paper_authors._id: 651b9345b9ae9f37afb52ef1
paper_authors.name: Tianjian Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-03T10:10:15.137Z
paper_authors._id: 6487a75fcc0e54f272bc8853
paper_authors.avatarUrl: /avatars/d6d140fc8a2fc23d3ff5c2d2bdc70829.svg
paper_authors.fullname: Tianjian Lucas Gao
paper_authors.type: user
paper_authors.user: RaTE-Paper
paper_authors._id: 651b9345b9ae9f37afb52ef2
paper_authors.name: Yuexin Wu
paper_authors._id: 651b9345b9ae9f37afb52ef3
paper_authors.name: Yunxuan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-03T10:10:45.744Z
paper_authors._id: 63531848fc9b7ad0a53fa564
paper_authors.avatarUrl: /avatars/d8df470349aa976741604921cb408440.svg
paper_authors.fullname: Yunxuan Li
paper_authors.type: user
paper_authors.user: eric008
paper_authors._id: 651b9345b9ae9f37afb52ef4
paper_authors.name: Hongkun Yu
paper_authors._id: 651b9345b9ae9f37afb52ef5
paper_authors.name: Heng Ji
paper_title: Enable Language Models to Implicitly Learn Self-Improvement From Data
paper_summary: Large Language Models (LLMs) have demonstrated remarkable capabilities in
  open-ended text generation tasks. However, the inherent open-ended nature of
  these tasks implies that there is always room for improvement in the quality of
  model responses. To address this challenge, various approaches have been
  proposed to enhance the performance of LLMs. There has been a growing focus on
  enabling LLMs to self-improve their response quality, thereby reducing the
  reliance on extensive human annotation efforts for collecting diverse and
  high-quality training data. Recently, prompting-based methods have been widely
  explored among self-improvement methods owing to their effectiveness,
  efficiency, and convenience. However, those methods usually require explicitly
  and thoroughly written rubrics as inputs to LLMs. It is expensive and
  challenging to manually derive and provide all necessary rubrics with a
  real-world complex goal for improvement (e.g., being more helpful and less
  harmful). To this end, we propose an ImPlicit Self-ImprovemenT (PIT) framework
  that implicitly learns the improvement goal from human preference data. PIT
  only requires preference data that are used to train reward models without
  extra human efforts. Specifically, we reformulate the training objective of
  reinforcement learning from human feedback (RLHF) -- instead of maximizing
  response quality for a given input, we maximize the quality gap of the response
  conditioned on a reference response. In this way, PIT is implicitly trained
  with the improvement goal of better aligning with human preferences.
  Experiments on two real-world datasets and one synthetic dataset show that our
  method significantly outperforms prompting-based methods.
paper_ai_keywords: Large Language Models (LLMs) open-ended text generation tasks self-improving prompting-based methods human preference data ImPlicit Self-ImprovemenT (PIT) framework reward models reinforcement learning from human feedback (RLHF) quality gap reference response better aligning with human preferences