title: Debate Helps Weak-to-Strong Generalization
summary: Common methods for aligning already-capable models with desired behavior rely
  on the ability of humans to provide supervision. However, future superhuman
  models will surpass the capability of humans. Therefore, humans will only be
  able to weakly supervise superhuman models. This expected deficiency of human
  evaluation would weaken the safety of future AI systems. Scalable oversight and
  weak-to-strong generalization are two complementary approaches to tackle this
  issue. In this paper, we attempt to combine the strengths of these two
  approaches to further improve alignment. Specifically, we investigate ways of
  improving human supervision with a strong pretrained model and then supervise
  the strong model with enhanced weak human supervision. To make iterative
  empirical progress, we consider an analogy: can we use a strong model to
  improve weak model supervision and then use it to supervise the strong model?
  We empirically test it by finetuning a small weak model on ground truth labels
  with the additional help from a large strong model, and then finetuning the
  strong model on labels generated by the weak model. We find that debate can
  assist a weak model in extracting trustworthy information from an untrustworthy
  strong model, which provides leverage as context on samples when training a
  weak model. We also show that an ensemble of weak models helps exploit long
  arguments generated by strong model debaters and obtain a more robust
  supervision estimate. Extensive experiments on the OpenAI weak-to-strong NLP
  benchmarks show that the combination approach leads to better alignment, which
  indicates that debate has the potential to help weak-to-strong generalization.
paper_authors._id: 6793188b56f015277a9ed95c
paper_authors.name: Hao Lang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:09:24.142Z
paper_authors._id: 65a6131fee7aa779f5bf8329
paper_authors.avatarUrl: /avatars/aa25cc3153fd7e511b51b801e8107564.svg
paper_authors.fullname: langhao
paper_authors.type: user
paper_authors.user: langnick
paper_authors._id: 6793188b56f015277a9ed95d
paper_authors.name: Fei Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:09:39.353Z
paper_authors._id: 635b8b6a37c6a2c12e2cce00
paper_authors.avatarUrl: /avatars/229fb72180529141515d1df797b33709.svg
paper_authors.fullname: Fei Huang
paper_authors.type: user
paper_authors.user: hzhwcmhf
paper_authors._id: 6793188b56f015277a9ed95e
paper_authors.name: Yongbin Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:08:59.394Z
paper_authors._id: 66641b2fd8e1e34bc621e688
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66641b2fd8e1e34bc621e688/csPETwnx2zCIHSWi9uAi-.png
paper_authors.fullname: Yongbin Li
paper_authors.type: user
paper_authors.user: Yongbin-Li
paper_title: Debate Helps Weak-to-Strong Generalization
paper_summary: Common methods for aligning already-capable models with desired behavior rely
  on the ability of humans to provide supervision. However, future superhuman
  models will surpass the capability of humans. Therefore, humans will only be
  able to weakly supervise superhuman models. This expected deficiency of human
  evaluation would weaken the safety of future AI systems. Scalable oversight and
  weak-to-strong generalization are two complementary approaches to tackle this
  issue. In this paper, we attempt to combine the strengths of these two
  approaches to further improve alignment. Specifically, we investigate ways of
  improving human supervision with a strong pretrained model and then supervise
  the strong model with enhanced weak human supervision. To make iterative
  empirical progress, we consider an analogy: can we use a strong model to
  improve weak model supervision and then use it to supervise the strong model?
  We empirically test it by finetuning a small weak model on ground truth labels
  with the additional help from a large strong model, and then finetuning the
  strong model on labels generated by the weak model. We find that debate can
  assist a weak model in extracting trustworthy information from an untrustworthy
  strong model, which provides leverage as context on samples when training a
  weak model. We also show that an ensemble of weak models helps exploit long
  arguments generated by strong model debaters and obtain a more robust
  supervision estimate. Extensive experiments on the OpenAI weak-to-strong NLP
  benchmarks show that the combination approach leads to better alignment, which
  indicates that debate has the potential to help weak-to-strong generalization.
paper_ai_summary: Debate between weak and strong models improves the alignment of AI systems by extracting trustworthy information and providing robust supervision estimates.
paper_ai_keywords: weak supervision superhuman models scalable oversight weak-to-strong generalization fine-tuning debate ensemble methods OpenAI weak-to-strong NLP benchmarks