title: MiCRo: Mixture Modeling and Context-aware Routing for Personalized
    Preference Learning
summary: Reward modeling is a key step in building safe foundation models when
  applying reinforcement learning from human feedback (RLHF) to align Large
  Language Models (LLMs). However, reward modeling based on the Bradley-Terry
  (BT) model assumes a global reward function, failing to capture the inherently
  diverse and heterogeneous human preferences. Hence, such oversimplification
  limits LLMs from supporting personalization and pluralistic alignment.
  Theoretically, we show that when human preferences follow a mixture
  distribution of diverse subgroups, a single BT model has an irreducible error.
  While existing solutions, such as multi-objective learning with fine-grained
  annotations, help address this issue, they are costly and constrained by
  predefined attributes, failing to fully capture the richness of human values.
  In this work, we introduce MiCRo, a two-stage framework that enhances
  personalized preference learning by leveraging large-scale binary preference
  datasets without requiring explicit fine-grained annotations. In the first
  stage, MiCRo introduces context-aware mixture modeling approach to capture
  diverse human preferences. In the second stage, MiCRo integrates an online
  routing strategy that dynamically adapts mixture weights based on specific
  context to resolve ambiguity, allowing for efficient and scalable preference
  adaptation with minimal additional supervision. Experiments on multiple
  preference datasets demonstrate that MiCRo effectively captures diverse human
  preferences and significantly improves downstream personalization.
paper_authors._id: 683e82f2fa7ede4842f95214
paper_authors.name: Jingyan Shen
paper_authors._id: 683e82f2fa7ede4842f95215
paper_authors.name: Jiarui Yao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T08:40:58.100Z
paper_authors._id: 66f8689725464a7989b75845
paper_authors.avatarUrl: /avatars/43a61a528c5779103eaf5687ba44ee14.svg
paper_authors.fullname: Jiarui Yao
paper_authors.type: user
paper_authors.user: FlippyDora
paper_authors._id: 683e82f2fa7ede4842f95216
paper_authors.name: Rui Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T08:40:16.572Z
paper_authors._id: 64d45451c34a346181b130dd
paper_authors.avatarUrl: /avatars/9bb8205b889337df5d321539c9b5d69d.svg
paper_authors.fullname: Rui Yang
paper_authors.type: user
paper_authors.user: Ray2333
paper_authors._id: 683e82f2fa7ede4842f95217
paper_authors.name: Yifan Sun
paper_authors._id: 683e82f2fa7ede4842f95218
paper_authors.name: Feng Luo
paper_authors._id: 683e82f2fa7ede4842f95219
paper_authors.name: Rui Pan
paper_authors._id: 683e82f2fa7ede4842f9521a
paper_authors.name: Tong Zhang
paper_authors._id: 683e82f2fa7ede4842f9521b
paper_authors.name: Han Zhao
paper_title: MiCRo: Mixture Modeling and Context-aware Routing for Personalized
    Preference Learning
paper_summary: Reward modeling is a key step in building safe foundation models when
  applying reinforcement learning from human feedback (RLHF) to align Large
  Language Models (LLMs). However, reward modeling based on the Bradley-Terry
  (BT) model assumes a global reward function, failing to capture the inherently
  diverse and heterogeneous human preferences. Hence, such oversimplification
  limits LLMs from supporting personalization and pluralistic alignment.
  Theoretically, we show that when human preferences follow a mixture
  distribution of diverse subgroups, a single BT model has an irreducible error.
  While existing solutions, such as multi-objective learning with fine-grained
  annotations, help address this issue, they are costly and constrained by
  predefined attributes, failing to fully capture the richness of human values.
  In this work, we introduce MiCRo, a two-stage framework that enhances
  personalized preference learning by leveraging large-scale binary preference
  datasets without requiring explicit fine-grained annotations. In the first
  stage, MiCRo introduces context-aware mixture modeling approach to capture
  diverse human preferences. In the second stage, MiCRo integrates an online
  routing strategy that dynamically adapts mixture weights based on specific
  context to resolve ambiguity, allowing for efficient and scalable preference
  adaptation with minimal additional supervision. Experiments on multiple
  preference datasets demonstrate that MiCRo effectively captures diverse human
  preferences and significantly improves downstream personalization.
paper_ai_summary: MiCRo, a two-stage framework, improves personalized preference learning for large language models by leveraging binary preference datasets and dynamically adapting mixture weights based on context, effectively capturing diverse human preferences.
paper_ai_keywords: Reward modeling reinforcement learning from human feedback (RLHF) Large Language Models (LLMs) Bradley-Terry (BT) model mixture distribution personalization pluralistic alignment multi-objective learning context-aware mixture modeling online routing strategy