DeAL: Decoding-time Alignment for Large Language Models
Large Language Models (LLMs) are nowadays expected to generate content
aligned with human preferences. Current work focuses on alignment at model
training time, through techniques such as Reinforcement Learning with Human
Feedback (RLHF). However, it is unclear if such methods are an effective choice
to teach alignment objectives to the model. First, the inability to incorporate
multiple, custom rewards and reliance on a model developer's view of universal
and static principles are key limitations. Second, the residual gaps in model
training and the reliability of such approaches are also questionable (e.g.
susceptibility to jail-breaking even after safety training). To address these,
we propose DeAL, a framework that allows the user to customize reward functions
and enables Decoding-time Alignment of LLMs (DeAL). At its core, we view
decoding as a heuristic-guided search process and facilitate the use of a wide
variety of alignment objectives. Our experiments with programmatic constraints
such as keyword and length constraints (studied widely in the pre-LLM era) and
abstract objectives such as harmlessness and helpfulness (proposed in the
post-LLM era) show that we can DeAL with fine-grained trade-offs, improve
adherence to alignment objectives, and address residual gaps in LLMs. Lastly,
while DeAL can be effectively paired with RLHF and prompting techniques, its
generality makes decoding slower, an optimization we leave for future work.
65c9887676da32869fce9e42
James Y. Huang
65c9887676da32869fce9e43
Sailik Sengupta
claimed_verified
2024-02-13T08:10:21.010Z
65caef6d19683f9817c54568
/avatars/2f7a745bd14ed7d1a096d5a95121ed38.svg
Sailik Sengupta
user
sailiks
65c9887676da32869fce9e44
Daniele Bonadiman
65c9887676da32869fce9e45
Yi-an Lai
admin_assigned
2024-02-12T16:59:22.969Z
657f3ea683543a061b7bd9a4
/avatars/22eb7dd4413c15abce9397a866b32d8a.svg
Yi-An Lai
user
yianlai
65c9887676da32869fce9e46
Arshit Gupta
65c9887676da32869fce9e47
Nikolaos Pappas
65c9887676da32869fce9e48
Saab Mansour
65c9887676da32869fce9e49
Katrin Kirchoff
65c9887676da32869fce9e4a
Dan Roth
DeAL: Decoding-time Alignment for Large Language Models
Large Language Models (LLMs) are nowadays expected to generate content
aligned with human preferences. Current work focuses on alignment at model
training time, through techniques such as Reinforcement Learning with Human
Feedback (RLHF). However, it is unclear if such methods are an effective choice
to teach alignment objectives to the model. First, the inability to incorporate
multiple, custom rewards and reliance on a model developer's view of universal
and static principles are key limitations. Second, the residual gaps in model
training and the reliability of such approaches are also questionable (e.g.
susceptibility to jail-breaking even after safety training). To address these,
we propose DeAL, a framework that allows the user to customize reward functions
and enables Decoding-time Alignment of LLMs (DeAL). At its core, we view
decoding as a heuristic-guided search process and facilitate the use of a wide
variety of alignment objectives. Our experiments with programmatic constraints
such as keyword and length constraints (studied widely in the pre-LLM era) and
abstract objectives such as harmlessness and helpfulness (proposed in the
post-LLM era) show that we can DeAL with fine-grained trade-offs, improve
adherence to alignment objectives, and address residual gaps in LLMs. Lastly,
while DeAL can be effectively paired with RLHF and prompting techniques, its
generality makes decoding slower, an optimization we leave for future work.
A framework called DeAL enhances decoding-time alignment for Large Language Models by allowing customizable reward functions and improving adherence to alignment objectives.
Reinforcement Learning with Human Feedback (RLHF) Large Language Models (LLMs) Decoding-time Alignment heuristic-guided search keyword constraints length constraints harmlessness helpfulness