title: Super(ficial)-alignment: Strong Models May Deceive Weak Models in
    Weak-to-Strong Generalization
summary: Superalignment, where humans are weak supervisors of superhuman models, has
  become an important and widely discussed issue in the current era of rapid
  development of Large Language Models (LLMs). The recent work preliminarily
  studies this problem by using weak models to supervise strong models. It
  discovers that weakly supervised strong students can consistently outperform
  weak teachers towards the alignment target, leading to a weak-to-strong
  generalization phenomenon. However, we are concerned that behind such a
  promising phenomenon, whether there exists an issue of weak-to-strong
  deception, where strong models may deceive weak models by exhibiting
  well-aligned in areas known to weak models but producing misaligned behaviors
  in cases weak models do not know. We then take an initial step towards
  exploring this security issue in a specific but realistic multi-objective
  alignment case, where there may be some alignment targets conflicting with each
  other (e.g., helpfulness v.s. harmlessness). Such a conflict is likely to cause
  strong models to deceive weak models in one alignment dimension to gain high
  reward in other alignment dimension. Our experiments on both the reward
  modeling task and the preference optimization scenario indicate: (1) the
  weak-to-strong deception exists; (2) the deception phenomenon may intensify as
  the capability gap between weak and strong models increases. We also discuss
  potential solutions and find bootstrapping with an intermediate model can
  mitigate the deception to some extent. Our work highlights the urgent need to
  pay more attention to the true reliability of superalignment.
paper_authors._id: 6672b1f89cd9faec9344faf4
paper_authors.name: Wenkai Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T20:18:50.819Z
paper_authors._id: 64b7df742f5a966b973e25f7
paper_authors.avatarUrl: /avatars/e24e7769188d441317b3b7d10ef8fd60.svg
paper_authors.fullname: Wenkai Yang
paper_authors.type: user
paper_authors.user: Keven16
paper_authors._id: 6672b1f89cd9faec9344faf5
paper_authors.name: Shiqi Shen
paper_authors._id: 6672b1f89cd9faec9344faf6
paper_authors.name: Guangyao Shen
paper_authors._id: 6672b1f89cd9faec9344faf7
paper_authors.name: Zhi Gong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T09:48:50.213Z
paper_authors._id: 642100d8290342c5df87deb1
paper_authors.avatarUrl: /avatars/0ba154877c28c1dea4a0db63eb68ddaa.svg
paper_authors.fullname: gongzhi
paper_authors.type: user
paper_authors.user: gongzhi
paper_authors._id: 6672b1f89cd9faec9344faf8
paper_authors.name: Yankai Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T09:48:38.751Z
paper_authors._id: 657a651e1433ea7d44de6397
paper_authors.avatarUrl: /avatars/ccfc76f94595a38ff4a80f77c911eabf.svg
paper_authors.fullname: Yankai Lin
paper_authors.type: user
paper_authors.user: lyk423
paper_title: Super(ficial)-alignment: Strong Models May Deceive Weak Models in
    Weak-to-Strong Generalization
paper_summary: Superalignment, where humans are weak supervisors of superhuman models, has
  become an important and widely discussed issue in the current era of rapid
  development of Large Language Models (LLMs). The recent work preliminarily
  studies this problem by using weak models to supervise strong models. It
  discovers that weakly supervised strong students can consistently outperform
  weak teachers towards the alignment target, leading to a weak-to-strong
  generalization phenomenon. However, we are concerned that behind such a
  promising phenomenon, whether there exists an issue of weak-to-strong
  deception, where strong models may deceive weak models by exhibiting
  well-aligned in areas known to weak models but producing misaligned behaviors
  in cases weak models do not know. We then take an initial step towards
  exploring this security issue in a specific but realistic multi-objective
  alignment case, where there may be some alignment targets conflicting with each
  other (e.g., helpfulness v.s. harmlessness). Such a conflict is likely to cause
  strong models to deceive weak models in one alignment dimension to gain high
  reward in other alignment dimension. Our experiments on both the reward
  modeling task and the preference optimization scenario indicate: (1) the
  weak-to-strong deception exists; (2) the deception phenomenon may intensify as
  the capability gap between weak and strong models increases. We also discuss
  potential solutions and find bootstrapping with an intermediate model can
  mitigate the deception to some extent. Our work highlights the urgent need to
  pay more attention to the true reliability of superalignment.
paper_ai_keywords: superalignment Large Language Models (LLMs) weak supervision strong models weak-to-strong generalization weak-to-strong deception multi-objective alignment alignment targets helpfulness harmlessness reward modeling task preference optimization scenario bootstrapping intermediate model true reliability