title: BOND: Aligning LLMs with Best-of-N Distillation
summary: Reinforcement learning from human feedback (RLHF) is a key driver of quality
  and safety in state-of-the-art large language models. Yet, a surprisingly
  simple and strong inference-time strategy is Best-of-N sampling that selects
  the best generation among N candidates. In this paper, we propose Best-of-N
  Distillation (BOND), a novel RLHF algorithm that seeks to emulate Best-of-N but
  without its significant computational overhead at inference time. Specifically,
  BOND is a distribution matching algorithm that forces the distribution of
  generations from the policy to get closer to the Best-of-N distribution. We use
  the Jeffreys divergence (a linear combination of forward and backward KL) to
  balance between mode-covering and mode-seeking behavior, and derive an
  iterative formulation that utilizes a moving anchor for efficiency. We
  demonstrate the effectiveness of our approach and several design choices
  through experiments on abstractive summarization and Gemma models. Aligning
  Gemma policies with BOND outperforms other RLHF algorithms by improving results
  on several benchmarks.
paper_authors._id: 669f50b24b821f5697c3a895
paper_authors.name: Pier Giuseppe Sessa
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T08:46:21.684Z
paper_authors._id: 667a7876becec8fc5112cf1f
paper_authors.avatarUrl: /avatars/6874f0e5a7edca4145c57d7908f89840.svg
paper_authors.fullname: Pier Giuseppe Sessa
paper_authors.type: user
paper_authors.user: piergs
paper_authors._id: 669f50b24b821f5697c3a896
paper_authors.name: Robert Dadashi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T08:46:27.635Z
paper_authors._id: 65fc7c0da57b88a765aea493
paper_authors.avatarUrl: /avatars/440f88769da1113d6158fe7e0514ead3.svg
paper_authors.fullname: Robert Dadashi
paper_authors.type: user
paper_authors.user: ddsh
paper_authors._id: 669f50b24b821f5697c3a897
paper_authors.name: Léonard Hussenot
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T08:46:36.305Z
paper_authors._id: 667a7b2e1764931987803b46
paper_authors.avatarUrl: /avatars/da2ee162f33c681e3d1c4d1e6d44dcbd.svg
paper_authors.fullname: Léonard Hussenot
paper_authors.type: user
paper_authors.user: leonardhussenot
paper_authors._id: 669f50b24b821f5697c3a898
paper_authors.name: Johan Ferret
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T08:46:42.102Z
paper_authors._id: 65afb7dbdd6bdfd73cd8e609
paper_authors.avatarUrl: /avatars/b21069bc2d7ee4cc1508008e3c8ade64.svg
paper_authors.fullname: Johan Ferret
paper_authors.type: user
paper_authors.user: ferretj
paper_authors._id: 669f50b24b821f5697c3a899
paper_authors.name: Nino Vieillard
paper_authors._id: 669f50b24b821f5697c3a89a
paper_authors.name: Alexandre Ramé
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T08:46:55.023Z
paper_authors._id: 63c94ede00104ea998de19a6
paper_authors.avatarUrl: /avatars/273959d87f0c67747588cf0700d64039.svg
paper_authors.fullname: Alexandre Rame
paper_authors.type: user
paper_authors.user: alexrame
paper_authors._id: 669f50b24b821f5697c3a89b
paper_authors.name: Bobak Shariari
paper_authors._id: 669f50b24b821f5697c3a89c
paper_authors.name: Sarah Perrin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T10:04:32.674Z
paper_authors._id: 66328157b270ae503e91339b
paper_authors.avatarUrl: /avatars/ea7a52060f5360f523ca28e137e85e33.svg
paper_authors.fullname: Sarah Perrin
paper_authors.type: user
paper_authors.user: Sper42
paper_authors._id: 669f50b24b821f5697c3a89d
paper_authors.name: Abe Friesen
paper_authors._id: 669f50b24b821f5697c3a89e
paper_authors.name: Geoffrey Cideron
paper_authors._id: 669f50b24b821f5697c3a89f
paper_authors.name: Sertan Girgin
paper_authors._id: 669f50b24b821f5697c3a8a0
paper_authors.name: Piotr Stanczyk
paper_authors._id: 669f50b24b821f5697c3a8a1
paper_authors.name: Andrea Michi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T10:08:27.975Z
paper_authors._id: 66466ef3b080e0f563dd2372
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66466ef3b080e0f563dd2372/wOMZgsd8r28iwAvqmYQRI.jpeg
paper_authors.fullname: Andrea Michi
paper_authors.type: user
paper_authors.user: andreamichi
paper_authors._id: 669f50b24b821f5697c3a8a2
paper_authors.name: Danila Sinopalnikov
paper_authors._id: 669f50b24b821f5697c3a8a3
paper_authors.name: Sabela Ramos
paper_authors._id: 669f50b24b821f5697c3a8a4
paper_authors.name: Amélie Héliou
paper_authors._id: 669f50b24b821f5697c3a8a5
paper_authors.name: Aliaksei Severyn
paper_authors._id: 669f50b24b821f5697c3a8a6
paper_authors.name: Matt Hoffman
paper_authors._id: 669f50b24b821f5697c3a8a7
paper_authors.name: Nikola Momchev
paper_authors._id: 669f50b24b821f5697c3a8a8
paper_authors.name: Olivier Bachem
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T10:04:40.875Z
paper_authors._id: 667aac871bffb68706b4f62c
paper_authors.avatarUrl: /avatars/2e2f2431f03a368f604c992d0d7ca57a.svg
paper_authors.fullname: Olivier Bachem
paper_authors.type: user
paper_authors.user: bachem
paper_title: BOND: Aligning LLMs with Best-of-N Distillation
paper_summary: Reinforcement learning from human feedback (RLHF) is a key driver of quality
  and safety in state-of-the-art large language models. Yet, a surprisingly
  simple and strong inference-time strategy is Best-of-N sampling that selects
  the best generation among N candidates. In this paper, we propose Best-of-N
  Distillation (BOND), a novel RLHF algorithm that seeks to emulate Best-of-N but
  without its significant computational overhead at inference time. Specifically,
  BOND is a distribution matching algorithm that forces the distribution of
  generations from the policy to get closer to the Best-of-N distribution. We use
  the Jeffreys divergence (a linear combination of forward and backward KL) to
  balance between mode-covering and mode-seeking behavior, and derive an
  iterative formulation that utilizes a moving anchor for efficiency. We
  demonstrate the effectiveness of our approach and several design choices
  through experiments on abstractive summarization and Gemma models. Aligning
  Gemma policies with BOND outperforms other RLHF algorithms by improving results
  on several benchmarks.
paper_ai_summary: BOND, a novel RLHF algorithm, distills the Best-of-N sampling strategy without significant computational overhead, enhancing the quality of generative language models.
paper_ai_keywords: Reinforcement learning from human feedback RLHF Best-of-N sampling Best-of-N Distillation BOND distribution matching Jeffreys divergence forward KL backward KL mode-covering mode-seeking abstractive summarization Gemma models