DogeRM: Equipping Reward Models with Domain Knowledge through Model Merging
<|reference_start|>DogeRM: Equipping Reward Models with Domain Knowledge through Model Merging: Reinforcement learning from human feedback (RLHF) is a popular strategy for aligning large language models (LLMs) with desired behaviors. Reward modeling is a crucial step in RLHF. However, collecting paired preference data for training reward models is often costly and time-consuming, especially for domain-specific preferences requiring expert annotation. To address this challenge, we propose the \textbf{Do}main knowled\textbf{ge} merged \textbf{R}eward \textbf{M}odel (DogeRM), a novel framework that integrates domain-specific knowledge into a general reward model by model merging. The experiments demonstrate that DogeRM enhances performance across different benchmarks and provide a detailed analysis showcasing the effects of model merging, showing the great potential of facilitating model alignment.<|reference_end|>
@article{lin2024dogerm:,
    title={DogeRM: Equipping Reward Models with Domain Knowledge through Model
  Merging},
    author={Tzu-Han Lin and Chen-An Li and Hung-yi Lee and Yun-Nung Chen},
    journal={arXiv preprint arXiv:2407.01470},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.01470},
    primaryClass={cs.CL}
}