title: Sample-Efficient Alignment for LLMs
summary: We study methods for efficiently aligning large language models (LLMs) with
  human preferences given budgeted online feedback. We first formulate the LLM
  alignment problem in the frame of contextual dueling bandits. This formulation,
  subsuming recent paradigms such as online RLHF and online DPO, inherently
  quests for sample-efficient algorithms that incorporate online active
  exploration. Leveraging insights from bandit theory, we introduce a unified
  algorithm based on Thompson sampling and highlight its applications in two
  distinct LLM alignment scenarios. The practical agent that efficiently
  implements this algorithm, named SEA (Sample-Efficient Alignment), is
  empirically validated through extensive experiments across three model scales
  (1B, 2.8B, 6.9B) and three preference learning algorithms (DPO, IPO, SLiC). The
  results demonstrate that SEA achieves highly sample-efficient alignment with
  oracle's preferences, outperforming recent active exploration methods for LLMs.
  Additionally, we release the implementation of SEA together with an efficient
  codebase designed for online alignment of LLMs, aiming to accelerate future
  research in this field.
paper_authors._id: 672ad09444b2453edfd339a7
paper_authors.name: Zichen Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:37:21.730Z
paper_authors._id: 65f5392c68b8e0cb3c9977a2
paper_authors.avatarUrl: /avatars/aa64772475098e8a135c13072fde6744.svg
paper_authors.fullname: Zichen
paper_authors.type: user
paper_authors.user: lkevinzc
paper_authors._id: 672ad09444b2453edfd339a8
paper_authors.name: Changyu Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:37:19.497Z
paper_authors._id: 64e416dc54e18f390ef79ba4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/5n01J00ZaVRrebsON8iYA.jpeg
paper_authors.fullname: Changyu Chen
paper_authors.type: user
paper_authors.user: Cameron-Chen
paper_authors._id: 672ad09444b2453edfd339a9
paper_authors.name: Chao Du
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:37:17.255Z
paper_authors._id: 632407c892e07e3ca20aca28
paper_authors.avatarUrl: /avatars/23b51b37b12b51a0947f687d1de4d3b5.svg
paper_authors.fullname: Chao Du
paper_authors.type: user
paper_authors.user: duchao
paper_authors._id: 672ad09444b2453edfd339aa
paper_authors.name: Wee Sun Lee
paper_authors._id: 672ad09444b2453edfd339ab
paper_authors.name: Min Lin
paper_title: Sample-Efficient Alignment for LLMs
paper_summary: We study methods for efficiently aligning large language models (LLMs) with
  human preferences given budgeted online feedback. We first formulate the LLM
  alignment problem in the frame of contextual dueling bandits. This formulation,
  subsuming recent paradigms such as online RLHF and online DPO, inherently
  quests for sample-efficient algorithms that incorporate online active
  exploration. Leveraging insights from bandit theory, we introduce a unified
  algorithm based on Thompson sampling and highlight its applications in two
  distinct LLM alignment scenarios. The practical agent that efficiently
  implements this algorithm, named SEA (Sample-Efficient Alignment), is
  empirically validated through extensive experiments across three model scales
  (1B, 2.8B, 6.9B) and three preference learning algorithms (DPO, IPO, SLiC). The
  results demonstrate that SEA achieves highly sample-efficient alignment with
  oracle's preferences, outperforming recent active exploration methods for LLMs.
  Additionally, we release the implementation of SEA together with an efficient
  codebase designed for online alignment of LLMs, aiming to accelerate future
  research in this field.
paper_ai_summary: A sample-efficient algorithm called SEA is introduced for aligning large language models with human preferences using contextual dueling bandits and Thompson sampling.
paper_ai_keywords: contextual dueling bandits online RLHF online DPO Thompson sampling sample-efficient active exploration DPO IPO SLiC online alignment