title: WorldPM: Scaling Human Preference Modeling
summary: Motivated by scaling laws in language modeling that demonstrate how test loss
  scales as a power law with model and dataset sizes, we find that similar laws
  exist in preference modeling. We propose World Preference Modeling$ (WorldPM)
  to emphasize this scaling potential, where World Preference embodies a unified
  representation of human preferences. In this paper, we collect preference data
  from public forums covering diverse user communities, and conduct extensive
  training using 15M-scale data across models ranging from 1.5B to 72B
  parameters. We observe distinct patterns across different evaluation metrics:
  (1) Adversarial metrics (ability to identify deceptive features) consistently
  scale up with increased training data and base model size; (2) Objective
  metrics (objective knowledge with well-defined answers) show emergent behavior
  in larger language models, highlighting WorldPM's scalability potential; (3)
  Subjective metrics (subjective preferences from a limited number of humans or
  AI) do not demonstrate scaling trends. Further experiments validate the
  effectiveness of WorldPM as a foundation for preference fine-tuning. Through
  evaluations on 7 benchmarks with 20 subtasks, we find that WorldPM broadly
  improves the generalization performance across human preference datasets of
  varying sizes (7K, 100K and 800K samples), with performance gains exceeding 5%
  on many key subtasks. Integrating WorldPM into our internal RLHF pipeline, we
  observe significant improvements on both in-house and public evaluation sets,
  with notable gains of 4% to 8% in our in-house evaluations.
paper_authors._id: 682699da19c4a596dbcea4f5
paper_authors.name: Binghai Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-16T13:24:15.616Z
paper_authors._id: 642127bbe40f66bcd1e7e047
paper_authors.avatarUrl: /avatars/8da3b0ae9723af8d96906c7307e90bb8.svg
paper_authors.fullname: wang binghai
paper_authors.type: user
paper_authors.user: refrain-wbh
paper_authors._id: 682699da19c4a596dbcea4f6
paper_authors.name: Runji Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-18T08:47:31.143Z
paper_authors._id: 649a52e5de0fb7f3f499e583
paper_authors.avatarUrl: /avatars/25f6106fa168ae57ad5cd8ef55c70d31.svg
paper_authors.fullname: Runji Lin
paper_authors.type: user
paper_authors.user: RunjiLin
paper_authors._id: 682699da19c4a596dbcea4f7
paper_authors.name: Keming Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-16T13:40:18.414Z
paper_authors._id: 6453fa96ed6d7fede94408e0
paper_authors.avatarUrl: /avatars/e8c9025ef24cec958c87a1008bb54fd7.svg
paper_authors.fullname: Keming Lu
paper_authors.type: user
paper_authors.user: keminglu
paper_authors._id: 682699da19c4a596dbcea4f8
paper_authors.name: Le Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-18T08:47:33.198Z
paper_authors._id: 6455056fe4952d1c6cb45e49
paper_authors.avatarUrl: /avatars/730f61ebc4f2df7489499ad7471a426e.svg
paper_authors.fullname: Le Yu
paper_authors.type: user
paper_authors.user: vanillaOVO
paper_authors._id: 682699da19c4a596dbcea4f9
paper_authors.name: Zhenru Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-16T13:40:11.604Z
paper_authors._id: 64704e973601bb7b06643e98
paper_authors.avatarUrl: /avatars/52e51f4d1be6769e4397b8be2799cf32.svg
paper_authors.fullname: Zhenru Zhang
paper_authors.type: user
paper_authors.user: Zhenru
paper_authors._id: 682699da19c4a596dbcea4fa
paper_authors.name: Fei Huang
paper_authors._id: 682699da19c4a596dbcea4fb
paper_authors.name: Chujie Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-16T13:24:17.629Z
paper_authors._id: 610b70452719facd4ea85e28
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/610b70452719facd4ea85e28/S7nMy7D0Rxq0VIVblhYDG.jpeg
paper_authors.fullname: Chujie Zheng
paper_authors.type: user
paper_authors.user: chujiezheng
paper_authors._id: 682699da19c4a596dbcea4fc
paper_authors.name: Kai Dang
paper_authors._id: 682699da19c4a596dbcea4fd
paper_authors.name: Yang Fan
paper_authors._id: 682699da19c4a596dbcea4fe
paper_authors.name: Xingzhang Ren
paper_authors._id: 682699da19c4a596dbcea4ff
paper_authors.name: An Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-19T06:43:06.615Z
paper_authors._id: 62088594a5943c8a8fc94560
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1644733028938-62088594a5943c8a8fc94560.png
paper_authors.fullname: An Yang
paper_authors.type: user
paper_authors.user: yangapku
paper_authors._id: 682699da19c4a596dbcea500
paper_authors.name: Binyuan Hui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-16T13:39:55.653Z
paper_authors._id: 61e4c4ca1ab24785ac11ba69
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61e4c4ca1ab24785ac11ba69/1Q1zhhyGSJ9RJG9MzwxVv.jpeg
paper_authors.fullname: Binyuan Hui
paper_authors.type: user
paper_authors.user: huybery
paper_authors._id: 682699da19c4a596dbcea501
paper_authors.name: Dayiheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-16T13:39:49.138Z
paper_authors._id: 6434d4989bd5a84b5dd0b0f5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6434d4989bd5a84b5dd0b0f5/0Elf9qbfG9Hkgypm9pTGm.jpeg
paper_authors.fullname: Dayiheng Liu
paper_authors.type: user
paper_authors.user: Losin94
paper_authors._id: 682699da19c4a596dbcea502
paper_authors.name: Tao Gui
paper_authors._id: 682699da19c4a596dbcea503
paper_authors.name: Qi Zhang
paper_authors._id: 682699da19c4a596dbcea504
paper_authors.name: Xuanjing Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-16T13:39:38.400Z
paper_authors._id: 67f9c4ee171948c38302ae0f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Cqb3ijr_sZkpLhEEEEybK.png
paper_authors.fullname: Xuanjing Huang
paper_authors.type: user
paper_authors.user: xjhuang
paper_authors._id: 682699da19c4a596dbcea505
paper_authors.name: Yu-Gang Jiang
paper_authors._id: 682699da19c4a596dbcea506
paper_authors.name: Bowen Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-16T13:39:20.178Z
paper_authors._id: 6583ab7983a9e1460c67d876
paper_authors.avatarUrl: /avatars/74400bc448c3f07e23a4cd53d68a6af7.svg
paper_authors.fullname: bowen
paper_authors.type: user
paper_authors.user: bowenYu
paper_authors._id: 682699da19c4a596dbcea507
paper_authors.name: Jingren Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-16T13:39:13.182Z
paper_authors._id: 602f88f5e8149a962412a667
paper_authors.avatarUrl: /avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Jingren
paper_authors._id: 682699da19c4a596dbcea508
paper_authors.name: Junyang Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-16T13:39:02.362Z
paper_authors._id: 620760a26e3b7210c2ff1943
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
paper_authors.fullname: Junyang Lin
paper_authors.type: user
paper_authors.user: JustinLin610
paper_title: WorldPM: Scaling Human Preference Modeling
paper_summary: Motivated by scaling laws in language modeling that demonstrate how test loss
  scales as a power law with model and dataset sizes, we find that similar laws
  exist in preference modeling. We propose World Preference Modeling$ (WorldPM)
  to emphasize this scaling potential, where World Preference embodies a unified
  representation of human preferences. In this paper, we collect preference data
  from public forums covering diverse user communities, and conduct extensive
  training using 15M-scale data across models ranging from 1.5B to 72B
  parameters. We observe distinct patterns across different evaluation metrics:
  (1) Adversarial metrics (ability to identify deceptive features) consistently
  scale up with increased training data and base model size; (2) Objective
  metrics (objective knowledge with well-defined answers) show emergent behavior
  in larger language models, highlighting WorldPM's scalability potential; (3)
  Subjective metrics (subjective preferences from a limited number of humans or
  AI) do not demonstrate scaling trends. Further experiments validate the
  effectiveness of WorldPM as a foundation for preference fine-tuning. Through
  evaluations on 7 benchmarks with 20 subtasks, we find that WorldPM broadly
  improves the generalization performance across human preference datasets of
  varying sizes (7K, 100K and 800K samples), with performance gains exceeding 5%
  on many key subtasks. Integrating WorldPM into our internal RLHF pipeline, we
  observe significant improvements on both in-house and public evaluation sets,
  with notable gains of 4% to 8% in our in-house evaluations.
paper_ai_summary: World Preference Modeling (WorldPM) enhances preference fine-tuning through large-scale preference data, showing scalability benefits in adversarial and objective metrics, and improving generalization across various human preference datasets.
paper_ai_keywords: World Preference Modeling preference data preference fine-tuning adversarial metrics objective metrics subjective metrics generalization performance RLHF pipeline