title: MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions
summary: Reinforcement learning from human feedback (RLHF) has demonstrated
  effectiveness in aligning large language models (LLMs) with human preferences.
  However, token-level RLHF suffers from the credit assignment problem over long
  sequences, where delayed rewards make it challenging for the model to discern
  which actions contributed to successful outcomes. This hinders learning
  efficiency and slows convergence. In this paper, we propose MA-RLHF, a simple
  yet effective RLHF framework that incorporates macro actions -- sequences of
  tokens or higher-level language constructs -- into the learning process. By
  operating at this higher level of abstraction, our approach reduces the
  temporal distance between actions and rewards, facilitating faster and more
  accurate credit assignment. This results in more stable policy gradient
  estimates and enhances learning efficiency within each episode, all without
  increasing computational complexity during training or inference. We validate
  our approach through extensive experiments across various model sizes and
  tasks, including text summarization, dialogue generation, question answering,
  and program synthesis. Our method achieves substantial performance improvements
  over standard RLHF, with performance gains of up to 30% in text summarization
  and code generation, 18% in dialogue, and 8% in question answering tasks.
  Notably, our approach reaches parity with vanilla RLHF 1.7x to 2x faster in
  terms of training time and continues to outperform it with further training. We
  will make our code and data publicly available at
  https://github.com/ernie-research/MA-RLHF .
paper_authors._id: 6705ff67f6d7020cce5daaab
paper_authors.name: Yekun Chai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-09T13:32:24.626Z
paper_authors._id: 626a6ce8adf559c88de9ace0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1651142275247-626a6ce8adf559c88de9ace0.jpeg
paper_authors.fullname: CYK
paper_authors.type: user
paper_authors.user: cyk1337
paper_authors._id: 6705ff67f6d7020cce5daaac
paper_authors.name: Haoran Sun
paper_authors._id: 6705ff67f6d7020cce5daaad
paper_authors.name: Huang Fang
paper_authors._id: 6705ff67f6d7020cce5daaae
paper_authors.name: Shuohuan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-09T12:58:15.521Z
paper_authors._id: 63202a4af7db36538c9fe3ba
paper_authors.avatarUrl: /avatars/d270cfb9c22183e4edc3a91bd12ce66a.svg
paper_authors.fullname: Shuohuan Wang
paper_authors.type: user
paper_authors.user: Fyzal
paper_authors._id: 6705ff67f6d7020cce5daaaf
paper_authors.name: Yu Sun
paper_authors._id: 6705ff67f6d7020cce5daab0
paper_authors.name: Hua Wu
paper_title: MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions
paper_summary: Reinforcement learning from human feedback (RLHF) has demonstrated
  effectiveness in aligning large language models (LLMs) with human preferences.
  However, token-level RLHF suffers from the credit assignment problem over long
  sequences, where delayed rewards make it challenging for the model to discern
  which actions contributed to successful outcomes. This hinders learning
  efficiency and slows convergence. In this paper, we propose MA-RLHF, a simple
  yet effective RLHF framework that incorporates macro actions -- sequences of
  tokens or higher-level language constructs -- into the learning process. By
  operating at this higher level of abstraction, our approach reduces the
  temporal distance between actions and rewards, facilitating faster and more
  accurate credit assignment. This results in more stable policy gradient
  estimates and enhances learning efficiency within each episode, all without
  increasing computational complexity during training or inference. We validate
  our approach through extensive experiments across various model sizes and
  tasks, including text summarization, dialogue generation, question answering,
  and program synthesis. Our method achieves substantial performance improvements
  over standard RLHF, with performance gains of up to 30% in text summarization
  and code generation, 18% in dialogue, and 8% in question answering tasks.
  Notably, our approach reaches parity with vanilla RLHF 1.7x to 2x faster in
  terms of training time and continues to outperform it with further training. We
  will make our code and data publicly available at
  https://github.com/ernie-research/MA-RLHF .
paper_ai_keywords: reinforcement learning from human feedback (RLHF) large language models (LLMs) token-level RLHF credit assignment problem macro actions higher-level language constructs policy gradient estimates text summarization dialogue generation question answering program synthesis