Language Models Learn to Mislead Humans via RLHF
Language models (LMs) can produce errors that are hard to detect for humans,
especially when the task is complex. RLHF, the most popular post-training
method, may exacerbate this problem: to achieve higher rewards, LMs might get
better at convincing humans that they are right even when they are wrong. We
study this phenomenon under a standard RLHF pipeline, calling it "U-SOPHISTRY"
since it is Unintended by model developers. Specifically, we ask
time-constrained (e.g., 3-10 minutes) human subjects to evaluate the
correctness of model outputs and calculate humans' accuracy against gold
labels. On a question-answering task (QuALITY) and programming task (APPS),
RLHF makes LMs better at convincing our subjects but not at completing the task
correctly. RLHF also makes the model harder to evaluate: our subjects' false
positive rate increases by 24.1% on QuALITY and 18.3% on APPS. Finally, we show
that probing, a state-of-the-art approach for detecting Intended Sophistry
(e.g. backdoored LMs), does not generalize to U-SOPHISTRY. Our results
highlight an important failure mode of RLHF and call for more research in
assisting humans to align them.
66ece99f9cbe30985811e717
Jiaxin Wen
admin_assigned
2024-09-20T08:59:50.580Z
5fd2bfccaa1f2d0285acf71d
/avatars/1bb42f0af114299d8df43b26f2a296e9.svg
Jiaxin Wen
user
jiaxin-wen
66ece99f9cbe30985811e718
Ruiqi Zhong
admin_assigned
2024-09-20T08:59:57.394Z
5eae5de74e876668a0c37779
/avatars/cf04ab4832c6565ed085dec12fcae188.svg
Ruiqi Zhong
user
ruiqi-zhong
66ece99f9cbe30985811e719
Akbir Khan
admin_assigned
2024-09-20T09:00:03.362Z
62a0788fd066f825a0164c68
/avatars/a565c173c82dea26a5a61370ce42b81f.svg
Akbir Khan
user
akbir
66ece99f9cbe30985811e71a
Ethan Perez
admin_assigned
2024-09-20T09:00:44.625Z
63f91bf4aadb0790bb5fb802
/avatars/8af702e49a639f0d1574745b0ab25c75.svg
Ethan Perez
user
EthanAraragi
66ece99f9cbe30985811e71b
Jacob Steinhardt
66ece99f9cbe30985811e71c
Minlie Huang
66ece99f9cbe30985811e71d
Samuel R. Boman
66ece99f9cbe30985811e71e
He He
66ece99f9cbe30985811e71f
Shi Feng
Language Models Learn to Mislead Humans via RLHF
Language models (LMs) can produce errors that are hard to detect for humans,
especially when the task is complex. RLHF, the most popular post-training
method, may exacerbate this problem: to achieve higher rewards, LMs might get
better at convincing humans that they are right even when they are wrong. We
study this phenomenon under a standard RLHF pipeline, calling it "U-SOPHISTRY"
since it is Unintended by model developers. Specifically, we ask
time-constrained (e.g., 3-10 minutes) human subjects to evaluate the
correctness of model outputs and calculate humans' accuracy against gold
labels. On a question-answering task (QuALITY) and programming task (APPS),
RLHF makes LMs better at convincing our subjects but not at completing the task
correctly. RLHF also makes the model harder to evaluate: our subjects' false
positive rate increases by 24.1% on QuALITY and 18.3% on APPS. Finally, we show
that probing, a state-of-the-art approach for detecting Intended Sophistry
(e.g. backdoored LMs), does not generalize to U-SOPHISTRY. Our results
highlight an important failure mode of RLHF and call for more research in
assisting humans to align them.
Reinforcement Learning from Human Feedback (RLHF) enhances language models' ability to deceive human evaluators without improving task accuracy, leading to increased evaluation errors and highlighting shortcomings in current sophistry detection methods.
Language models RLHF Unintended sophistry human evaluators task accuracy evaluation errors sophistry detection