Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs
In this report, we introduce a collection of methods to enhance reward
modeling for LLMs, focusing specifically on data-centric techniques. We propose
effective data selection and filtering strategies for curating high-quality
open-source preference datasets, culminating in the Skywork-Reward data
collection, which contains only 80K preference pairs -- significantly smaller
than existing datasets. Using this curated dataset, we developed the
Skywork-Reward model series -- Skywork-Reward-Gemma-27B and
Skywork-Reward-Llama-3.1-8B -- with the former currently holding the top
position on the RewardBench leaderboard. Notably, our techniques and datasets
have directly enhanced the performance of many top-ranked models on
RewardBench, highlighting the practical impact of our contributions in
real-world preference learning applications.
671af231aa3e4ccb7b12e259
Chris Yuhao Liu
claimed_verified
2024-10-25T09:30:26.746Z
658229ef5f6d83438257fce5
/avatars/b4417de9a338e95dc69cc547a46348e8.svg
Chris (Yuhao) Liu
user
chrisliu298
671af231aa3e4ccb7b12e25a
Liang Zeng
claimed_verified
2024-10-25T09:30:23.740Z
6621efe1a6eec3ad03e38759
/avatars/c35acce69f244ec0833dffd53eedf6a3.svg
Liang Zeng
user
zengliangcs
671af231aa3e4ccb7b12e25b
Jiacai Liu
claimed_verified
2025-01-26T11:39:57.628Z
65643645b09c0b9ece1b8f0e
/avatars/d5197103b6e92f765bfda7ed2cc8d53e.svg
Jiacai Liu
user
skydownacai
671af231aa3e4ccb7b12e25c
Rui Yan
671af231aa3e4ccb7b12e25d
Jujie He
671af231aa3e4ccb7b12e25e
Chaojie Wang
671af231aa3e4ccb7b12e25f
Shuicheng Yan
671af231aa3e4ccb7b12e260
Yang Liu
671af231aa3e4ccb7b12e261
Yahui Zhou
Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs
In this report, we introduce a collection of methods to enhance reward
modeling for LLMs, focusing specifically on data-centric techniques. We propose
effective data selection and filtering strategies for curating high-quality
open-source preference datasets, culminating in the Skywork-Reward data
collection, which contains only 80K preference pairs -- significantly smaller
than existing datasets. Using this curated dataset, we developed the
Skywork-Reward model series -- Skywork-Reward-Gemma-27B and
Skywork-Reward-Llama-3.1-8B -- with the former currently holding the top
position on the RewardBench leaderboard. Notably, our techniques and datasets
have directly enhanced the performance of many top-ranked models on
RewardBench, highlighting the practical impact of our contributions in
real-world preference learning applications.
Curated preference datasets and selection strategies enhanced reward models, leading to improved performance on the RewardBench leaderboard.
Kywork-Reward data selection filtering strategies Skywork-Reward-Gemma-27B Skywork-Reward-Llama-3.1-8B RewardBench