title: Rethinking Diverse Human Preference Learning through Principal Component
    Analysis
summary: Understanding human preferences is crucial for improving foundation models
  and building personalized AI systems. However, preferences are inherently
  diverse and complex, making it difficult for traditional reward models to
  capture their full range. While fine-grained preference data can help,
  collecting it is expensive and hard to scale. In this paper, we introduce
  Decomposed Reward Models (DRMs), a novel approach that extracts diverse human
  preferences from binary comparisons without requiring fine-grained annotations.
  Our key insight is to represent human preferences as vectors and analyze them
  using Principal Component Analysis (PCA). By constructing a dataset of
  embedding differences between preferred and rejected responses, DRMs identify
  orthogonal basis vectors that capture distinct aspects of preference. These
  decomposed rewards can be flexibly combined to align with different user needs,
  offering an interpretable and scalable alternative to traditional reward
  models. We demonstrate that DRMs effectively extract meaningful preference
  dimensions (e.g., helpfulness, safety, humor) and adapt to new users without
  additional training. Our results highlight DRMs as a powerful framework for
  personalized and interpretable LLM alignment.
paper_authors._id: 67b5461d29cc269e5a4eb823
paper_authors.name: Feng Luo
paper_authors._id: 67b5461d29cc269e5a4eb824
paper_authors.name: Rui Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T09:01:23.095Z
paper_authors._id: 64d45451c34a346181b130dd
paper_authors.avatarUrl: /avatars/9bb8205b889337df5d321539c9b5d69d.svg
paper_authors.fullname: Rui Yang
paper_authors.type: user
paper_authors.user: Ray2333
paper_authors._id: 67b5461d29cc269e5a4eb825
paper_authors.name: Hao Sun
paper_authors._id: 67b5461d29cc269e5a4eb826
paper_authors.name: Chunyuan Deng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T09:56:33.053Z
paper_authors._id: 634b9914dcf125e4da02498b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/634b9914dcf125e4da02498b/crRgFroWq5U6XWtvlTXSZ.jpeg
paper_authors.fullname: Chunyuan Deng
paper_authors.type: user
paper_authors.user: CharlesDDDD
paper_authors._id: 67b5461d29cc269e5a4eb827
paper_authors.name: Jiarui Yao
paper_authors._id: 67b5461d29cc269e5a4eb828
paper_authors.name: Jingyan Shen
paper_authors._id: 67b5461d29cc269e5a4eb829
paper_authors.name: Huan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T09:52:47.329Z
paper_authors._id: 6719d581a6cad13741b8bc7f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6719d581a6cad13741b8bc7f/w4EttqfXRgWZJc6HpYOS9.jpeg
paper_authors.fullname: Huan Zhang
paper_authors.type: user
paper_authors.user: huanzhang12
paper_authors._id: 67b5461d29cc269e5a4eb82a
paper_authors.name: Hanjie Chen
paper_title: Rethinking Diverse Human Preference Learning through Principal Component
    Analysis
paper_summary: Understanding human preferences is crucial for improving foundation models
  and building personalized AI systems. However, preferences are inherently
  diverse and complex, making it difficult for traditional reward models to
  capture their full range. While fine-grained preference data can help,
  collecting it is expensive and hard to scale. In this paper, we introduce
  Decomposed Reward Models (DRMs), a novel approach that extracts diverse human
  preferences from binary comparisons without requiring fine-grained annotations.
  Our key insight is to represent human preferences as vectors and analyze them
  using Principal Component Analysis (PCA). By constructing a dataset of
  embedding differences between preferred and rejected responses, DRMs identify
  orthogonal basis vectors that capture distinct aspects of preference. These
  decomposed rewards can be flexibly combined to align with different user needs,
  offering an interpretable and scalable alternative to traditional reward
  models. We demonstrate that DRMs effectively extract meaningful preference
  dimensions (e.g., helpfulness, safety, humor) and adapt to new users without
  additional training. Our results highlight DRMs as a powerful framework for
  personalized and interpretable LLM alignment.
paper_ai_keywords: Decomposed Reward Models (DRMs) binary comparisons Principal Component Analysis (PCA) embedding differences orthogonal basis vectors preference dimensions personalized and interpretable LLM alignment