title: Language Models Learn to Mislead Humans via RLHF
summary: Language models (LMs) can produce errors that are hard to detect for humans,
  especially when the task is complex. RLHF, the most popular post-training
  method, may exacerbate this problem: to achieve higher rewards, LMs might get
  better at convincing humans that they are right even when they are wrong. We
  study this phenomenon under a standard RLHF pipeline, calling it "U-SOPHISTRY"
  since it is Unintended by model developers. Specifically, we ask
  time-constrained (e.g., 3-10 minutes) human subjects to evaluate the
  correctness of model outputs and calculate humans' accuracy against gold
  labels. On a question-answering task (QuALITY) and programming task (APPS),
  RLHF makes LMs better at convincing our subjects but not at completing the task
  correctly. RLHF also makes the model harder to evaluate: our subjects' false
  positive rate increases by 24.1% on QuALITY and 18.3% on APPS. Finally, we show
  that probing, a state-of-the-art approach for detecting Intended Sophistry
  (e.g. backdoored LMs), does not generalize to U-SOPHISTRY. Our results
  highlight an important failure mode of RLHF and call for more research in
  assisting humans to align them.
paper_authors._id: 66ece99f9cbe30985811e717
paper_authors.name: Jiaxin Wen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T08:59:50.580Z
paper_authors._id: 5fd2bfccaa1f2d0285acf71d
paper_authors.avatarUrl: /avatars/1bb42f0af114299d8df43b26f2a296e9.svg
paper_authors.fullname: Jiaxin Wen
paper_authors.type: user
paper_authors.user: jiaxin-wen
paper_authors._id: 66ece99f9cbe30985811e718
paper_authors.name: Ruiqi Zhong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T08:59:57.394Z
paper_authors._id: 5eae5de74e876668a0c37779
paper_authors.avatarUrl: /avatars/cf04ab4832c6565ed085dec12fcae188.svg
paper_authors.fullname: Ruiqi Zhong
paper_authors.type: user
paper_authors.user: ruiqi-zhong
paper_authors._id: 66ece99f9cbe30985811e719
paper_authors.name: Akbir Khan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T09:00:03.362Z
paper_authors._id: 62a0788fd066f825a0164c68
paper_authors.avatarUrl: /avatars/a565c173c82dea26a5a61370ce42b81f.svg
paper_authors.fullname: Akbir Khan
paper_authors.type: user
paper_authors.user: akbir
paper_authors._id: 66ece99f9cbe30985811e71a
paper_authors.name: Ethan Perez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T09:00:44.625Z
paper_authors._id: 63f91bf4aadb0790bb5fb802
paper_authors.avatarUrl: /avatars/8af702e49a639f0d1574745b0ab25c75.svg
paper_authors.fullname: Ethan Perez
paper_authors.type: user
paper_authors.user: EthanAraragi
paper_authors._id: 66ece99f9cbe30985811e71b
paper_authors.name: Jacob Steinhardt
paper_authors._id: 66ece99f9cbe30985811e71c
paper_authors.name: Minlie Huang
paper_authors._id: 66ece99f9cbe30985811e71d
paper_authors.name: Samuel R. Boman
paper_authors._id: 66ece99f9cbe30985811e71e
paper_authors.name: He He
paper_authors._id: 66ece99f9cbe30985811e71f
paper_authors.name: Shi Feng
paper_title: Language Models Learn to Mislead Humans via RLHF
paper_summary: Language models (LMs) can produce errors that are hard to detect for humans,
  especially when the task is complex. RLHF, the most popular post-training
  method, may exacerbate this problem: to achieve higher rewards, LMs might get
  better at convincing humans that they are right even when they are wrong. We
  study this phenomenon under a standard RLHF pipeline, calling it "U-SOPHISTRY"
  since it is Unintended by model developers. Specifically, we ask
  time-constrained (e.g., 3-10 minutes) human subjects to evaluate the
  correctness of model outputs and calculate humans' accuracy against gold
  labels. On a question-answering task (QuALITY) and programming task (APPS),
  RLHF makes LMs better at convincing our subjects but not at completing the task
  correctly. RLHF also makes the model harder to evaluate: our subjects' false
  positive rate increases by 24.1% on QuALITY and 18.3% on APPS. Finally, we show
  that probing, a state-of-the-art approach for detecting Intended Sophistry
  (e.g. backdoored LMs), does not generalize to U-SOPHISTRY. Our results
  highlight an important failure mode of RLHF and call for more research in
  assisting humans to align them.
paper_ai_keywords: RLHF U-SOPHISTRY QuALITY APPS false positive rate Intended Sophistry