WorldPM: Scaling Human Preference Modeling
Motivated by scaling laws in language modeling that demonstrate how test loss
scales as a power law with model and dataset sizes, we find that similar laws
exist in preference modeling. We propose World Preference Modeling$ (WorldPM)
to emphasize this scaling potential, where World Preference embodies a unified
representation of human preferences. In this paper, we collect preference data
from public forums covering diverse user communities, and conduct extensive
training using 15M-scale data across models ranging from 1.5B to 72B
parameters. We observe distinct patterns across different evaluation metrics:
(1) Adversarial metrics (ability to identify deceptive features) consistently
scale up with increased training data and base model size; (2) Objective
metrics (objective knowledge with well-defined answers) show emergent behavior
in larger language models, highlighting WorldPM's scalability potential; (3)
Subjective metrics (subjective preferences from a limited number of humans or
AI) do not demonstrate scaling trends. Further experiments validate the
effectiveness of WorldPM as a foundation for preference fine-tuning. Through
evaluations on 7 benchmarks with 20 subtasks, we find that WorldPM broadly
improves the generalization performance across human preference datasets of
varying sizes (7K, 100K and 800K samples), with performance gains exceeding 5%
on many key subtasks. Integrating WorldPM into our internal RLHF pipeline, we
observe significant improvements on both in-house and public evaluation sets,
with notable gains of 4% to 8% in our in-house evaluations.
682699da19c4a596dbcea4f5
Binghai Wang
claimed_verified
2025-05-16T13:24:15.616Z
642127bbe40f66bcd1e7e047
/avatars/8da3b0ae9723af8d96906c7307e90bb8.svg
wang binghai
user
refrain-wbh
682699da19c4a596dbcea4f6
Runji Lin
claimed_verified
2025-05-18T08:47:31.143Z
649a52e5de0fb7f3f499e583
/avatars/25f6106fa168ae57ad5cd8ef55c70d31.svg
Runji Lin
user
RunjiLin
682699da19c4a596dbcea4f7
Keming Lu
admin_assigned
2025-05-16T13:40:18.414Z
6453fa96ed6d7fede94408e0
/avatars/e8c9025ef24cec958c87a1008bb54fd7.svg
Keming Lu
user
keminglu
682699da19c4a596dbcea4f8
Le Yu
claimed_verified
2025-05-18T08:47:33.198Z
6455056fe4952d1c6cb45e49
/avatars/bbdcd4405497f11ff3774352277d1ec7.svg
Le Yu
user
vanillaOVO
682699da19c4a596dbcea4f9
Zhenru Zhang
admin_assigned
2025-05-16T13:40:11.604Z
64704e973601bb7b06643e98
/avatars/52e51f4d1be6769e4397b8be2799cf32.svg
Zhenru Zhang
user
Zhenru
682699da19c4a596dbcea4fa
Fei Huang
682699da19c4a596dbcea4fb
Chujie Zheng
claimed_verified
2025-05-16T13:24:17.629Z
610b70452719facd4ea85e28
https://cdn-avatars.huggingface.co/v1/production/uploads/610b70452719facd4ea85e28/S7nMy7D0Rxq0VIVblhYDG.jpeg
Chujie Zheng
user
chujiezheng
682699da19c4a596dbcea4fc
Kai Dang
682699da19c4a596dbcea4fd
Yang Fan
682699da19c4a596dbcea4fe
Xingzhang Ren
682699da19c4a596dbcea4ff
An Yang
claimed_verified
2025-05-19T06:43:06.615Z
62088594a5943c8a8fc94560
https://cdn-avatars.huggingface.co/v1/production/uploads/1644733028938-62088594a5943c8a8fc94560.png
An Yang
user
yangapku
682699da19c4a596dbcea500
Binyuan Hui
admin_assigned
2025-05-16T13:39:55.653Z
61e4c4ca1ab24785ac11ba69
https://cdn-avatars.huggingface.co/v1/production/uploads/61e4c4ca1ab24785ac11ba69/1Q1zhhyGSJ9RJG9MzwxVv.jpeg
Binyuan Hui
user
huybery
682699da19c4a596dbcea501
Dayiheng Liu
admin_assigned
2025-05-16T13:39:49.138Z
6434d4989bd5a84b5dd0b0f5
https://cdn-avatars.huggingface.co/v1/production/uploads/6434d4989bd5a84b5dd0b0f5/0Elf9qbfG9Hkgypm9pTGm.jpeg
Dayiheng Liu
user
Losin94
682699da19c4a596dbcea502
Tao Gui
682699da19c4a596dbcea503
Qi Zhang
682699da19c4a596dbcea504
Xuanjing Huang
admin_assigned
2025-05-16T13:39:38.400Z
67f9c4ee171948c38302ae0f
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Cqb3ijr_sZkpLhEEEEybK.png
Xuanjing Huang
user
xjhuang
682699da19c4a596dbcea505
Yu-Gang Jiang
682699da19c4a596dbcea506
Bowen Yu
admin_assigned
2025-05-16T13:39:20.178Z
6583ab7983a9e1460c67d876
/avatars/74400bc448c3f07e23a4cd53d68a6af7.svg
bowen
user
bowenYu
682699da19c4a596dbcea507
Jingren Zhou
admin_assigned
2025-05-16T13:39:13.182Z
602f88f5e8149a962412a667
/avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
Zhou
user
Jingren
682699da19c4a596dbcea508
Junyang Lin
admin_assigned
2025-05-16T13:39:02.362Z
620760a26e3b7210c2ff1943
https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
Junyang Lin
user
JustinLin610
WorldPM: Scaling Human Preference Modeling
Motivated by scaling laws in language modeling that demonstrate how test loss
scales as a power law with model and dataset sizes, we find that similar laws
exist in preference modeling. We propose World Preference Modeling$ (WorldPM)
to emphasize this scaling potential, where World Preference embodies a unified
representation of human preferences. In this paper, we collect preference data
from public forums covering diverse user communities, and conduct extensive
training using 15M-scale data across models ranging from 1.5B to 72B
parameters. We observe distinct patterns across different evaluation metrics:
(1) Adversarial metrics (ability to identify deceptive features) consistently
scale up with increased training data and base model size; (2) Objective
metrics (objective knowledge with well-defined answers) show emergent behavior
in larger language models, highlighting WorldPM's scalability potential; (3)
Subjective metrics (subjective preferences from a limited number of humans or
AI) do not demonstrate scaling trends. Further experiments validate the
effectiveness of WorldPM as a foundation for preference fine-tuning. Through
evaluations on 7 benchmarks with 20 subtasks, we find that WorldPM broadly
improves the generalization performance across human preference datasets of
varying sizes (7K, 100K and 800K samples), with performance gains exceeding 5%
on many key subtasks. Integrating WorldPM into our internal RLHF pipeline, we
observe significant improvements on both in-house and public evaluation sets,
with notable gains of 4% to 8% in our in-house evaluations.
World Preference Modeling (WorldPM) enhances preference fine-tuning through large-scale preference data, showing scalability benefits in adversarial and objective metrics, and improving generalization across various human preference datasets.
World Preference Modeling preference data preference fine-tuning adversarial metrics objective metrics subjective metrics generalization performance RLHF pipeline