title: Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak
    Supervision
summary: Widely used alignment techniques, such as reinforcement learning from human
  feedback (RLHF), rely on the ability of humans to supervise model behavior -
  for example, to evaluate whether a model faithfully followed instructions or
  generated safe outputs. However, future superhuman models will behave in
  complex ways too difficult for humans to reliably evaluate; humans will only be
  able to weakly supervise superhuman models. We study an analogy to this
  problem: can weak model supervision elicit the full capabilities of a much
  stronger model? We test this using a range of pretrained language models in the
  GPT-4 family on natural language processing (NLP), chess, and reward modeling
  tasks. We find that when we naively finetune strong pretrained models on labels
  generated by a weak model, they consistently perform better than their weak
  supervisors, a phenomenon we call weak-to-strong generalization. However, we
  are still far from recovering the full capabilities of strong models with naive
  finetuning alone, suggesting that techniques like RLHF may scale poorly to
  superhuman models without further work. We find that simple methods can often
  significantly improve weak-to-strong generalization: for example, when
  finetuning GPT-4 with a GPT-2-level supervisor and an auxiliary confidence
  loss, we can recover close to GPT-3.5-level performance on NLP tasks. Our
  results suggest that it is feasible to make empirical progress today on a
  fundamental challenge of aligning superhuman models.
paper_authors._id: 657fa5f6e50ca9a69909621f
paper_authors.name: Collin Burns
paper_authors._id: 657fa5f6e50ca9a699096220
paper_authors.name: Pavel Izmailov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-18T09:29:21.544Z
paper_authors._id: 6348d3d60672b21524f983d6
paper_authors.avatarUrl: /avatars/935d93551c0bbd8a0cbfeaef4721206f.svg
paper_authors.fullname: Pavel Izmailov
paper_authors.type: user
paper_authors.user: izmailovpavel
paper_authors._id: 657fa5f6e50ca9a699096221
paper_authors.name: Jan Hendrik Kirchner
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-18T09:30:04.154Z
paper_authors._id: 63d1918df7f31a66a2d38140
paper_authors.avatarUrl: /avatars/17fd871735fc8c4ede5d4faab9295778.svg
paper_authors.fullname: Jan Hendrik Kirchner
paper_authors.type: user
paper_authors.user: jkirchner
paper_authors._id: 657fa5f6e50ca9a699096222
paper_authors.name: Bowen Baker
paper_authors._id: 657fa5f6e50ca9a699096223
paper_authors.name: Leo Gao
paper_authors._id: 657fa5f6e50ca9a699096224
paper_authors.name: Leopold Aschenbrenner
paper_authors._id: 657fa5f6e50ca9a699096225
paper_authors.name: Yining Chen
paper_authors._id: 657fa5f6e50ca9a699096226
paper_authors.name: Adrien Ecoffet
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-18T09:31:01.937Z
paper_authors._id: 649fbee82bf2e21d9562b606
paper_authors.avatarUrl: /avatars/92d593304a5f24875a573ee9bd725008.svg
paper_authors.fullname: Adrien Ecoffet
paper_authors.type: user
paper_authors.user: AdrienLE
paper_authors._id: 657fa5f6e50ca9a699096227
paper_authors.name: Manas Joglekar
paper_authors._id: 657fa5f6e50ca9a699096228
paper_authors.name: Jan Leike
paper_authors._id: 657fa5f6e50ca9a699096229
paper_authors.name: Ilya Sutskever
paper_authors._id: 657fa5f6e50ca9a69909622a
paper_authors.name: Jeff Wu
paper_title: Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak
    Supervision
paper_summary: Widely used alignment techniques, such as reinforcement learning from human
  feedback (RLHF), rely on the ability of humans to supervise model behavior -
  for example, to evaluate whether a model faithfully followed instructions or
  generated safe outputs. However, future superhuman models will behave in
  complex ways too difficult for humans to reliably evaluate; humans will only be
  able to weakly supervise superhuman models. We study an analogy to this
  problem: can weak model supervision elicit the full capabilities of a much
  stronger model? We test this using a range of pretrained language models in the
  GPT-4 family on natural language processing (NLP), chess, and reward modeling
  tasks. We find that when we naively finetune strong pretrained models on labels
  generated by a weak model, they consistently perform better than their weak
  supervisors, a phenomenon we call weak-to-strong generalization. However, we
  are still far from recovering the full capabilities of strong models with naive
  finetuning alone, suggesting that techniques like RLHF may scale poorly to
  superhuman models without further work. We find that simple methods can often
  significantly improve weak-to-strong generalization: for example, when
  finetuning GPT-4 with a GPT-2-level supervisor and an auxiliary confidence
  loss, we can recover close to GPT-3.5-level performance on NLP tasks. Our
  results suggest that it is feasible to make empirical progress today on a
  fundamental challenge of aligning superhuman models.
paper_ai_keywords: reinforcement learning from human feedback (RLHF) model supervision weak supervision pretrained language models GPT-4 natural language processing (NLP) chess reward modeling weak-to-strong generalization naive finetuning auxiliary confidence loss GPT-2-level supervisor GPT-3.5 system performance