title: Deep Bayesian Active Learning for Preference Modeling in Large Language
    Models
summary: Leveraging human preferences for steering the behavior of Large Language
  Models (LLMs) has demonstrated notable success in recent years. Nonetheless,
  data selection and labeling are still a bottleneck for these systems,
  particularly at large scale. Hence, selecting the most informative points for
  acquiring human feedback may considerably reduce the cost of preference
  labeling and unleash the further development of LLMs. Bayesian Active Learning
  provides a principled framework for addressing this challenge and has
  demonstrated remarkable success in diverse settings. However, previous attempts
  to employ it for Preference Modeling did not meet such expectations. In this
  work, we identify that naive epistemic uncertainty estimation leads to the
  acquisition of redundant samples. We address this by proposing the Bayesian
  Active Learner for Preference Modeling (BAL-PM), a novel stochastic acquisition
  policy that not only targets points of high epistemic uncertainty according to
  the preference model but also seeks to maximize the entropy of the acquired
  prompt distribution in the feature space spanned by the employed LLM. Notably,
  our experiments demonstrate that BAL-PM requires 33% to 68% fewer preference
  labels in two popular human preference datasets and exceeds previous stochastic
  Bayesian acquisition policies.
paper_authors._id: 6671b1cf059c8f7fc78187a5
paper_authors.name: Luckeciano C. Melo
paper_authors._id: 6671b1cf059c8f7fc78187a6
paper_authors.name: Panagiotis Tigas
paper_authors._id: 6671b1cf059c8f7fc78187a7
paper_authors.name: Alessandro Abate
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T10:30:27.816Z
paper_authors._id: 646723426192d3914221997b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/TL7QhvPJEMEch3a6f8cEr.jpeg
paper_authors.fullname: Alessandro Abate
paper_authors.type: user
paper_authors.user: SteamDrake
paper_authors._id: 6671b1cf059c8f7fc78187a8
paper_authors.name: Yarin Gal
paper_title: Deep Bayesian Active Learning for Preference Modeling in Large Language
    Models
paper_summary: Leveraging human preferences for steering the behavior of Large Language
  Models (LLMs) has demonstrated notable success in recent years. Nonetheless,
  data selection and labeling are still a bottleneck for these systems,
  particularly at large scale. Hence, selecting the most informative points for
  acquiring human feedback may considerably reduce the cost of preference
  labeling and unleash the further development of LLMs. Bayesian Active Learning
  provides a principled framework for addressing this challenge and has
  demonstrated remarkable success in diverse settings. However, previous attempts
  to employ it for Preference Modeling did not meet such expectations. In this
  work, we identify that naive epistemic uncertainty estimation leads to the
  acquisition of redundant samples. We address this by proposing the Bayesian
  Active Learner for Preference Modeling (BAL-PM), a novel stochastic acquisition
  policy that not only targets points of high epistemic uncertainty according to
  the preference model but also seeks to maximize the entropy of the acquired
  prompt distribution in the feature space spanned by the employed LLM. Notably,
  our experiments demonstrate that BAL-PM requires 33% to 68% fewer preference
  labels in two popular human preference datasets and exceeds previous stochastic
  Bayesian acquisition policies.
paper_ai_summary: The Bayesian Active Learner for Preference Modeling (BAL-PM) reduces the need for human preference labels by strategically selecting informative samples, outperforming previous methods.
paper_ai_keywords: Large Language Models (LLMs) Bayesian Active Learning preference modeling epistemic uncertainty entropy feature space preference labels