MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions
Reinforcement learning from human feedback (RLHF) has demonstrated
effectiveness in aligning large language models (LLMs) with human preferences.
However, token-level RLHF suffers from the credit assignment problem over long
sequences, where delayed rewards make it challenging for the model to discern
which actions contributed to successful outcomes. This hinders learning
efficiency and slows convergence. In this paper, we propose MA-RLHF, a simple
yet effective RLHF framework that incorporates macro actions -- sequences of
tokens or higher-level language constructs -- into the learning process. By
operating at this higher level of abstraction, our approach reduces the
temporal distance between actions and rewards, facilitating faster and more
accurate credit assignment. This results in more stable policy gradient
estimates and enhances learning efficiency within each episode, all without
increasing computational complexity during training or inference. We validate
our approach through extensive experiments across various model sizes and
tasks, including text summarization, dialogue generation, question answering,
and program synthesis. Our method achieves substantial performance improvements
over standard RLHF, with performance gains of up to 30% in text summarization
and code generation, 18% in dialogue, and 8% in question answering tasks.
Notably, our approach reaches parity with vanilla RLHF 1.7x to 2x faster in
terms of training time and continues to outperform it with further training. We
will make our code and data publicly available at
https://github.com/ernie-research/MA-RLHF .
6705ff67f6d7020cce5daaab
Yekun Chai
claimed_verified
2024-10-09T13:32:24.626Z
626a6ce8adf559c88de9ace0
https://cdn-avatars.huggingface.co/v1/production/uploads/1651142275247-626a6ce8adf559c88de9ace0.jpeg
CYK
user
cyk1337
6705ff67f6d7020cce5daaac
Haoran Sun
claimed_verified
2025-06-09T10:14:52.525Z
63b52bd4bda8d44adf2c802a
/avatars/4de7e7196f95fa53bf03af79b7197f96.svg
Haoran Sun
user
Moyu-hrsun
6705ff67f6d7020cce5daaad
Huang Fang
6705ff67f6d7020cce5daaae
Shuohuan Wang
admin_assigned
2024-10-09T12:58:15.521Z
63202a4af7db36538c9fe3ba
/avatars/d270cfb9c22183e4edc3a91bd12ce66a.svg
Shuohuan Wang
user
wangshuohuan
6705ff67f6d7020cce5daaaf
Yu Sun
6705ff67f6d7020cce5daab0
Hua Wu
MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions
Reinforcement learning from human feedback (RLHF) has demonstrated
effectiveness in aligning large language models (LLMs) with human preferences.
However, token-level RLHF suffers from the credit assignment problem over long
sequences, where delayed rewards make it challenging for the model to discern
which actions contributed to successful outcomes. This hinders learning
efficiency and slows convergence. In this paper, we propose MA-RLHF, a simple
yet effective RLHF framework that incorporates macro actions -- sequences of
tokens or higher-level language constructs -- into the learning process. By
operating at this higher level of abstraction, our approach reduces the
temporal distance between actions and rewards, facilitating faster and more
accurate credit assignment. This results in more stable policy gradient
estimates and enhances learning efficiency within each episode, all without
increasing computational complexity during training or inference. We validate
our approach through extensive experiments across various model sizes and
tasks, including text summarization, dialogue generation, question answering,
and program synthesis. Our method achieves substantial performance improvements
over standard RLHF, with performance gains of up to 30% in text summarization
and code generation, 18% in dialogue, and 8% in question answering tasks.
Notably, our approach reaches parity with vanilla RLHF 1.7x to 2x faster in
terms of training time and continues to outperform it with further training. We
will make our code and data publicly available at
https://github.com/ernie-research/MA-RLHF .
MA-RLHF improves reinforcement learning from human feedback by using macro actions to facilitate faster and more accurate credit assignment, enhancing learning efficiency across various tasks without increasing computational complexity.
reinforcement learning from human feedback RLHF large language models token-level RLHF credit assignment problem macro actions policy gradient estimates text summarization dialogue generation question answering program synthesis