title: Fast Best-of-N Decoding via Speculative Rejection
summary: The safe and effective deployment of Large Language Models (LLMs) involves a
  critical step called alignment, which ensures that the model's responses are in
  accordance with human preferences. Prevalent alignment techniques, such as DPO,
  PPO and their variants, align LLMs by changing the pre-trained model weights
  during a phase called post-training. While predominant, these post-training
  methods add substantial complexity before LLMs can be deployed. Inference-time
  alignment methods avoid the complex post-training step and instead bias the
  generation towards responses that are aligned with human preferences. The
  best-known inference-time alignment method, called Best-of-N, is as effective
  as the state-of-the-art post-training procedures. Unfortunately, Best-of-N
  requires vastly more resources at inference time than standard decoding
  strategies, which makes it computationally not viable. In this work, we
  introduce Speculative Rejection, a computationally-viable inference-time
  alignment algorithm. It generates high-scoring responses according to a given
  reward model, like Best-of-N does, while being between 16 to 32 times more
  computationally efficient.
paper_authors._id: 6720872c71d10b163b13e299
paper_authors.name: Hanshi Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:53:08.123Z
paper_authors._id: 651b23a0a6e00a1678c9d8af
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/651b23a0a6e00a1678c9d8af/_Ip-ysvcjGpFCI9HtcTgx.jpeg
paper_authors.fullname: Hanshi
paper_authors.type: user
paper_authors.user: preminstrel
paper_authors._id: 6720872c71d10b163b13e29a
paper_authors.name: Momin Haider
paper_authors._id: 6720872c71d10b163b13e29b
paper_authors.name: Ruiqi Zhang
paper_authors._id: 6720872c71d10b163b13e29c
paper_authors.name: Huitao Yang
paper_authors._id: 6720872c71d10b163b13e29d
paper_authors.name: Jiahao Qiu
paper_authors._id: 6720872c71d10b163b13e29e
paper_authors.name: Ming Yin
paper_authors._id: 6720872c71d10b163b13e29f
paper_authors.name: Mengdi Wang
paper_authors._id: 6720872c71d10b163b13e2a0
paper_authors.name: Peter Bartlett
paper_authors._id: 6720872c71d10b163b13e2a1
paper_authors.name: Andrea Zanette
paper_title: Fast Best-of-N Decoding via Speculative Rejection
paper_summary: The safe and effective deployment of Large Language Models (LLMs) involves a
  critical step called alignment, which ensures that the model's responses are in
  accordance with human preferences. Prevalent alignment techniques, such as DPO,
  PPO and their variants, align LLMs by changing the pre-trained model weights
  during a phase called post-training. While predominant, these post-training
  methods add substantial complexity before LLMs can be deployed. Inference-time
  alignment methods avoid the complex post-training step and instead bias the
  generation towards responses that are aligned with human preferences. The
  best-known inference-time alignment method, called Best-of-N, is as effective
  as the state-of-the-art post-training procedures. Unfortunately, Best-of-N
  requires vastly more resources at inference time than standard decoding
  strategies, which makes it computationally not viable. In this work, we
  introduce Speculative Rejection, a computationally-viable inference-time
  alignment algorithm. It generates high-scoring responses according to a given
  reward model, like Best-of-N does, while being between 16 to 32 times more
  computationally efficient.
paper_ai_keywords: Large Language Models (LLMs) alignment DPO PPO post-training inference-time alignment Best-of-N reward model Speculative Rejection