title: DeAL: Decoding-time Alignment for Large Language Models
summary: Large Language Models (LLMs) are nowadays expected to generate content
  aligned with human preferences. Current work focuses on alignment at model
  training time, through techniques such as Reinforcement Learning with Human
  Feedback (RLHF). However, it is unclear if such methods are an effective choice
  to teach alignment objectives to the model. First, the inability to incorporate
  multiple, custom rewards and reliance on a model developer's view of universal
  and static principles are key limitations. Second, the residual gaps in model
  training and the reliability of such approaches are also questionable (e.g.
  susceptibility to jail-breaking even after safety training). To address these,
  we propose DeAL, a framework that allows the user to customize reward functions
  and enables Decoding-time Alignment of LLMs (DeAL). At its core, we view
  decoding as a heuristic-guided search process and facilitate the use of a wide
  variety of alignment objectives. Our experiments with programmatic constraints
  such as keyword and length constraints (studied widely in the pre-LLM era) and
  abstract objectives such as harmlessness and helpfulness (proposed in the
  post-LLM era) show that we can DeAL with fine-grained trade-offs, improve
  adherence to alignment objectives, and address residual gaps in LLMs. Lastly,
  while DeAL can be effectively paired with RLHF and prompting techniques, its
  generality makes decoding slower, an optimization we leave for future work.
paper_authors._id: 65c9887676da32869fce9e42
paper_authors.name: James Y. Huang
paper_authors._id: 65c9887676da32869fce9e43
paper_authors.name: Sailik Sengupta
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-13T08:10:21.010Z
paper_authors._id: 65caef6d19683f9817c54568
paper_authors.avatarUrl: /avatars/2f7a745bd14ed7d1a096d5a95121ed38.svg
paper_authors.fullname: Sailik Sengupta
paper_authors.type: user
paper_authors.user: sailiks
paper_authors._id: 65c9887676da32869fce9e44
paper_authors.name: Daniele Bonadiman
paper_authors._id: 65c9887676da32869fce9e45
paper_authors.name: Yi-an Lai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T16:59:22.969Z
paper_authors._id: 657f3ea683543a061b7bd9a4
paper_authors.avatarUrl: /avatars/22eb7dd4413c15abce9397a866b32d8a.svg
paper_authors.fullname: Yi-An Lai
paper_authors.type: user
paper_authors.user: yianlai
paper_authors._id: 65c9887676da32869fce9e46
paper_authors.name: Arshit Gupta
paper_authors._id: 65c9887676da32869fce9e47
paper_authors.name: Nikolaos Pappas
paper_authors._id: 65c9887676da32869fce9e48
paper_authors.name: Saab Mansour
paper_authors._id: 65c9887676da32869fce9e49
paper_authors.name: Katrin Kirchoff
paper_authors._id: 65c9887676da32869fce9e4a
paper_authors.name: Dan Roth
paper_title: DeAL: Decoding-time Alignment for Large Language Models
paper_summary: Large Language Models (LLMs) are nowadays expected to generate content
  aligned with human preferences. Current work focuses on alignment at model
  training time, through techniques such as Reinforcement Learning with Human
  Feedback (RLHF). However, it is unclear if such methods are an effective choice
  to teach alignment objectives to the model. First, the inability to incorporate
  multiple, custom rewards and reliance on a model developer's view of universal
  and static principles are key limitations. Second, the residual gaps in model
  training and the reliability of such approaches are also questionable (e.g.
  susceptibility to jail-breaking even after safety training). To address these,
  we propose DeAL, a framework that allows the user to customize reward functions
  and enables Decoding-time Alignment of LLMs (DeAL). At its core, we view
  decoding as a heuristic-guided search process and facilitate the use of a wide
  variety of alignment objectives. Our experiments with programmatic constraints
  such as keyword and length constraints (studied widely in the pre-LLM era) and
  abstract objectives such as harmlessness and helpfulness (proposed in the
  post-LLM era) show that we can DeAL with fine-grained trade-offs, improve
  adherence to alignment objectives, and address residual gaps in LLMs. Lastly,
  while DeAL can be effectively paired with RLHF and prompting techniques, its
  generality makes decoding slower, an optimization we leave for future work.
paper_ai_keywords: Reinforcement Learning with Human Feedback (RLHF) DeAL Decoding-time Alignment heuristic-guided search process keyword constraints length constraints harmlessness helpfulness