title: Reward Steering with Evolutionary Heuristics for Decoding-time Alignment
summary: The widespread applicability and increasing omnipresence of LLMs have
  instigated a need to align LLM responses to user and stakeholder preferences.
  Many preference optimization approaches have been proposed that fine-tune LLM
  parameters to achieve good alignment. However, such parameter tuning is known
  to interfere with model performance on many tasks. Moreover, keeping up with
  shifting user preferences is tricky in such a situation. Decoding-time
  alignment with reward model guidance solves these issues at the cost of
  increased inference time. However, most of such methods fail to strike the
  right balance between exploration and exploitation of reward -- often due to
  the conflated formulation of these two aspects - to give well-aligned
  responses. To remedy this we decouple these two aspects and implement them in
  an evolutionary fashion: exploration is enforced by decoding from mutated
  instructions and exploitation is represented as the periodic replacement of
  poorly-rewarded generations with well-rewarded ones. Empirical evidences
  indicate that this strategy outperforms many preference optimization and
  decode-time alignment approaches on two widely accepted alignment benchmarks
  AlpacaEval 2 and MT-Bench. Our implementation will be available at:
  https://darwin-alignment.github.io.
paper_authors._id: 6678f8029f2810b009e0463a
paper_authors.name: Chia-Yu Hung
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-24T07:27:34.077Z
paper_authors._id: 62864cdf011a13fe5f84d746
paper_authors.avatarUrl: /avatars/d01d3f2261572a625949e6b9dee2aa47.svg
paper_authors.fullname: Hung Chia Yu
paper_authors.type: user
paper_authors.user: hungchiayu
paper_authors._id: 6678f8029f2810b009e0463b
paper_authors.name: Navonil Majumder
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-24T13:43:09.664Z
paper_authors._id: 644e20f267c9458c912bd246
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/5j7LZ-6sCarsEvKr2tPAj.png
paper_authors.fullname: Navonil Majumder
paper_authors.type: user
paper_authors.user: hollownight
paper_authors._id: 6678f8029f2810b009e0463c
paper_authors.name: Ambuj Mehrish
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-24T13:43:15.789Z
paper_authors._id: 6350a5170f376d3c482c9738
paper_authors.avatarUrl: /avatars/1dcf14c35a5a2018879c9c94b3ebe345.svg
paper_authors.fullname: Ambuj Mehrish
paper_authors.type: user
paper_authors.user: ambujm22
paper_authors._id: 6678f8029f2810b009e0463d
paper_authors.name: Soujanya Poria
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-24T13:10:35.414Z
paper_authors._id: 626b626405fe1cb65725aca1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/626b626405fe1cb65725aca1/aa-Lata46I3fXOmMetvXH.jpeg
paper_authors.fullname: Soujanya Poria
paper_authors.type: user
paper_authors.user: soujanyaporia
paper_title: Reward Steering with Evolutionary Heuristics for Decoding-time Alignment
paper_summary: The widespread applicability and increasing omnipresence of LLMs have
  instigated a need to align LLM responses to user and stakeholder preferences.
  Many preference optimization approaches have been proposed that fine-tune LLM
  parameters to achieve good alignment. However, such parameter tuning is known
  to interfere with model performance on many tasks. Moreover, keeping up with
  shifting user preferences is tricky in such a situation. Decoding-time
  alignment with reward model guidance solves these issues at the cost of
  increased inference time. However, most of such methods fail to strike the
  right balance between exploration and exploitation of reward -- often due to
  the conflated formulation of these two aspects - to give well-aligned
  responses. To remedy this we decouple these two aspects and implement them in
  an evolutionary fashion: exploration is enforced by decoding from mutated
  instructions and exploitation is represented as the periodic replacement of
  poorly-rewarded generations with well-rewarded ones. Empirical evidences
  indicate that this strategy outperforms many preference optimization and
  decode-time alignment approaches on two widely accepted alignment benchmarks
  AlpacaEval 2 and MT-Bench. Our implementation will be available at:
  https://darwin-alignment.github.io.
paper_ai_keywords: large language models (LLMs) parameter tuning decoding-time alignment reward model exploration exploitation Shapley values evolutionary fashion mutated instructions AlpacaEval 2 MT-Bench