title: DogeRM: Equipping Reward Models with Domain Knowledge through Model
    Merging
summary: Reinforcement learning from human feedback (RLHF) is a popular strategy for
  aligning large language models (LLMs) with desired behaviors. Reward modeling
  is a crucial step in RLHF. However, collecting paired preference data for
  training reward models is often costly and time-consuming, especially for
  domain-specific preferences requiring expert annotation. To address this
  challenge, we propose the Domain knowledge merged
  Reward Model (DogeRM), a novel framework that integrates
  domain-specific knowledge into a general reward model by model merging. The
  experiments demonstrate that DogeRM enhances performance across different
  benchmarks and provide a detailed analysis showcasing the effects of model
  merging, showing the great potential of facilitating model alignment.
paper_authors._id: 6683648cacdf070b0e255757
paper_authors.name: Tzu-Han Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-02T07:39:02.988Z
paper_authors._id: 6602787b1827b6d37ee527be
paper_authors.avatarUrl: /avatars/f69c1779b4347a042dad1a0d962145af.svg
paper_authors.fullname: Tzu-Han Lin
paper_authors.type: user
paper_authors.user: hank0316
paper_authors._id: 6683648cacdf070b0e255758
paper_authors.name: Chen-An Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-04T08:05:00.275Z
paper_authors._id: 64206697b286e8c464f50440
paper_authors.avatarUrl: /avatars/d628a269fcc4b572ed5868135e10f65a.svg
paper_authors.fullname: Chen An, Li
paper_authors.type: user
paper_authors.user: lca0503
paper_authors._id: 6683648cacdf070b0e255759
paper_authors.name: Hung-yi Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:08:52.318Z
paper_authors._id: 629b6eb3ad498ab19282ae6f
paper_authors.avatarUrl: /avatars/adade8c512122120b06e5c50724a178d.svg
paper_authors.fullname: Hung-yi Lee
paper_authors.type: user
paper_authors.user: hungyilee
paper_authors._id: 6683648cacdf070b0e25575a
paper_authors.name: Yun-Nung Chen
paper_title: DogeRM: Equipping Reward Models with Domain Knowledge through Model
    Merging
paper_summary: Reinforcement learning from human feedback (RLHF) is a popular strategy for
  aligning large language models (LLMs) with desired behaviors. Reward modeling
  is a crucial step in RLHF. However, collecting paired preference data for
  training reward models is often costly and time-consuming, especially for
  domain-specific preferences requiring expert annotation. To address this
  challenge, we propose the Domain knowledge merged
  Reward Model (DogeRM), a novel framework that integrates
  domain-specific knowledge into a general reward model by model merging. The
  experiments demonstrate that DogeRM enhances performance across different
  benchmarks and provide a detailed analysis showcasing the effects of model
  merging, showing the great potential of facilitating model alignment.
paper_ai_summary: DogeRM, a novel framework, integrates domain-specific knowledge to enhance reward models in reinforcement learning from human feedback, improving model alignment across benchmarks.
paper_ai_keywords: reinforcement learning from human feedback large language models reward modeling domain-specific preferences expert annotation model merging domain knowledge merged reward model DogeRM model alignment