title: Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs
summary: In this report, we introduce a collection of methods to enhance reward
  modeling for LLMs, focusing specifically on data-centric techniques. We propose
  effective data selection and filtering strategies for curating high-quality
  open-source preference datasets, culminating in the Skywork-Reward data
  collection, which contains only 80K preference pairs -- significantly smaller
  than existing datasets. Using this curated dataset, we developed the
  Skywork-Reward model series -- Skywork-Reward-Gemma-27B and
  Skywork-Reward-Llama-3.1-8B -- with the former currently holding the top
  position on the RewardBench leaderboard. Notably, our techniques and datasets
  have directly enhanced the performance of many top-ranked models on
  RewardBench, highlighting the practical impact of our contributions in
  real-world preference learning applications.
paper_authors._id: 671af231aa3e4ccb7b12e259
paper_authors.name: Chris Yuhao Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-25T09:30:26.746Z
paper_authors._id: 658229ef5f6d83438257fce5
paper_authors.avatarUrl: /avatars/b4417de9a338e95dc69cc547a46348e8.svg
paper_authors.fullname: Chris (Yuhao) Liu
paper_authors.type: user
paper_authors.user: chrisliu298
paper_authors._id: 671af231aa3e4ccb7b12e25a
paper_authors.name: Liang Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-25T09:30:23.740Z
paper_authors._id: 6621efe1a6eec3ad03e38759
paper_authors.avatarUrl: /avatars/c35acce69f244ec0833dffd53eedf6a3.svg
paper_authors.fullname: Liang Zeng
paper_authors.type: user
paper_authors.user: zengliangcs
paper_authors._id: 671af231aa3e4ccb7b12e25b
paper_authors.name: Jiacai Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-26T11:39:57.628Z
paper_authors._id: 65643645b09c0b9ece1b8f0e
paper_authors.avatarUrl: /avatars/d5197103b6e92f765bfda7ed2cc8d53e.svg
paper_authors.fullname: Jiacai Liu
paper_authors.type: user
paper_authors.user: skydownacai
paper_authors._id: 671af231aa3e4ccb7b12e25c
paper_authors.name: Rui Yan
paper_authors._id: 671af231aa3e4ccb7b12e25d
paper_authors.name: Jujie He
paper_authors._id: 671af231aa3e4ccb7b12e25e
paper_authors.name: Chaojie Wang
paper_authors._id: 671af231aa3e4ccb7b12e25f
paper_authors.name: Shuicheng Yan
paper_authors._id: 671af231aa3e4ccb7b12e260
paper_authors.name: Yang Liu
paper_authors._id: 671af231aa3e4ccb7b12e261
paper_authors.name: Yahui Zhou
paper_title: Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs
paper_summary: In this report, we introduce a collection of methods to enhance reward
  modeling for LLMs, focusing specifically on data-centric techniques. We propose
  effective data selection and filtering strategies for curating high-quality
  open-source preference datasets, culminating in the Skywork-Reward data
  collection, which contains only 80K preference pairs -- significantly smaller
  than existing datasets. Using this curated dataset, we developed the
  Skywork-Reward model series -- Skywork-Reward-Gemma-27B and
  Skywork-Reward-Llama-3.1-8B -- with the former currently holding the top
  position on the RewardBench leaderboard. Notably, our techniques and datasets
  have directly enhanced the performance of many top-ranked models on
  RewardBench, highlighting the practical impact of our contributions in
  real-world preference learning applications.
paper_ai_keywords: reward modeling LLMs data-centric techniques data selection data filtering high-quality open-source preference datasets Skywork-Reward data collection preference pairs Skywork-Reward model series Skywork-Reward-Gemma-27B Skywork-Reward-Llama-3.1-8B RewardBench preference learning applications