Enable Language Models to Implicitly Learn Self-Improvement From Data
Large Language Models (LLMs) have demonstrated remarkable capabilities in
open-ended text generation tasks. However, the inherent open-ended nature of
these tasks implies that there is always room for improvement in the quality of
model responses. To address this challenge, various approaches have been
proposed to enhance the performance of LLMs. There has been a growing focus on
enabling LLMs to self-improve their response quality, thereby reducing the
reliance on extensive human annotation efforts for collecting diverse and
high-quality training data. Recently, prompting-based methods have been widely
explored among self-improvement methods owing to their effectiveness,
efficiency, and convenience. However, those methods usually require explicitly
and thoroughly written rubrics as inputs to LLMs. It is expensive and
challenging to manually derive and provide all necessary rubrics with a
real-world complex goal for improvement (e.g., being more helpful and less
harmful). To this end, we propose an ImPlicit Self-ImprovemenT (PIT) framework
that implicitly learns the improvement goal from human preference data. PIT
only requires preference data that are used to train reward models without
extra human efforts. Specifically, we reformulate the training objective of
reinforcement learning from human feedback (RLHF) -- instead of maximizing
response quality for a given input, we maximize the quality gap of the response
conditioned on a reference response. In this way, PIT is implicitly trained
with the improvement goal of better aligning with human preferences.
Experiments on two real-world datasets and one synthetic dataset show that our
method significantly outperforms prompting-based methods.
651b9345b9ae9f37afb52eef
Ziqi Wang
extracted_confirmed
2023-10-03T10:11:23.704Z
651c321c4e15c57e1b0efca6
/avatars/241c4d975356bd986bad7303e1534bce.svg
Ziqi wang
user
wzq016
651b9345b9ae9f37afb52ef0
Le Hou
admin_assigned
2023-10-03T10:10:02.555Z
626b2402c804c432c1b5004c
/avatars/a119644422335e260af0ca4ef6399870.svg
Le Hou
user
solooobear
651b9345b9ae9f37afb52ef1
Tianjian Lu
admin_assigned
2023-10-03T10:10:15.137Z
6487a75fcc0e54f272bc8853
/avatars/d6d140fc8a2fc23d3ff5c2d2bdc70829.svg
Tianjian Lucas Gao
user
RaTE-Paper
651b9345b9ae9f37afb52ef2
Yuexin Wu
651b9345b9ae9f37afb52ef3
Yunxuan Li
admin_assigned
2023-10-03T10:10:45.744Z
63531848fc9b7ad0a53fa564
/avatars/d8df470349aa976741604921cb408440.svg
Yunxuan Li
user
eric008
651b9345b9ae9f37afb52ef4
Hongkun Yu
651b9345b9ae9f37afb52ef5
Heng Ji
Enable Language Models to Implicitly Learn Self-Improvement From Data
Large Language Models (LLMs) have demonstrated remarkable capabilities in
open-ended text generation tasks. However, the inherent open-ended nature of
these tasks implies that there is always room for improvement in the quality of
model responses. To address this challenge, various approaches have been
proposed to enhance the performance of LLMs. There has been a growing focus on
enabling LLMs to self-improve their response quality, thereby reducing the
reliance on extensive human annotation efforts for collecting diverse and
high-quality training data. Recently, prompting-based methods have been widely
explored among self-improvement methods owing to their effectiveness,
efficiency, and convenience. However, those methods usually require explicitly
and thoroughly written rubrics as inputs to LLMs. It is expensive and
challenging to manually derive and provide all necessary rubrics with a
real-world complex goal for improvement (e.g., being more helpful and less
harmful). To this end, we propose an ImPlicit Self-ImprovemenT (PIT) framework
that implicitly learns the improvement goal from human preference data. PIT
only requires preference data that are used to train reward models without
extra human efforts. Specifically, we reformulate the training objective of
reinforcement learning from human feedback (RLHF) -- instead of maximizing
response quality for a given input, we maximize the quality gap of the response
conditioned on a reference response. In this way, PIT is implicitly trained
with the improvement goal of better aligning with human preferences.
Experiments on two real-world datasets and one synthetic dataset show that our
method significantly outperforms prompting-based methods.
A new framework, ImPlicit Self-ImprovemenT (PIT), uses human preference data to implicitly enhance the quality of Large Language Model responses, outperforming prompting-based methods.
Large Language Models open-ended text generation self-improvement prompting-based methods reward models reinforcement learning from human feedback quality gap human preferences