MiCRo: Mixture Modeling and Context-aware Routing for Personalized
  Preference Learning
Reward modeling is a key step in building safe foundation models when
applying reinforcement learning from human feedback (RLHF) to align Large
Language Models (LLMs). However, reward modeling based on the Bradley-Terry
(BT) model assumes a global reward function, failing to capture the inherently
diverse and heterogeneous human preferences. Hence, such oversimplification
limits LLMs from supporting personalization and pluralistic alignment.
Theoretically, we show that when human preferences follow a mixture
distribution of diverse subgroups, a single BT model has an irreducible error.
While existing solutions, such as multi-objective learning with fine-grained
annotations, help address this issue, they are costly and constrained by
predefined attributes, failing to fully capture the richness of human values.
In this work, we introduce MiCRo, a two-stage framework that enhances
personalized preference learning by leveraging large-scale binary preference
datasets without requiring explicit fine-grained annotations. In the first
stage, MiCRo introduces context-aware mixture modeling approach to capture
diverse human preferences. In the second stage, MiCRo integrates an online
routing strategy that dynamically adapts mixture weights based on specific
context to resolve ambiguity, allowing for efficient and scalable preference
adaptation with minimal additional supervision. Experiments on multiple
preference datasets demonstrate that MiCRo effectively captures diverse human
preferences and significantly improves downstream personalization.
683e82f2fa7ede4842f95214
Jingyan Shen
683e82f2fa7ede4842f95215
Jiarui Yao
claimed_verified
2025-06-03T08:40:58.100Z
66f8689725464a7989b75845
/avatars/43a61a528c5779103eaf5687ba44ee14.svg
Jiarui Yao
user
FlippyDora
683e82f2fa7ede4842f95216
Rui Yang
claimed_verified
2025-06-03T08:40:16.572Z
64d45451c34a346181b130dd
/avatars/9bb8205b889337df5d321539c9b5d69d.svg
Rui Yang
user
Ray2333
683e82f2fa7ede4842f95217
Yifan Sun
683e82f2fa7ede4842f95218
Feng Luo
683e82f2fa7ede4842f95219
Rui Pan
683e82f2fa7ede4842f9521a
Tong Zhang
683e82f2fa7ede4842f9521b
Han Zhao
MiCRo: Mixture Modeling and Context-aware Routing for Personalized
  Preference Learning
Reward modeling is a key step in building safe foundation models when
applying reinforcement learning from human feedback (RLHF) to align Large
Language Models (LLMs). However, reward modeling based on the Bradley-Terry
(BT) model assumes a global reward function, failing to capture the inherently
diverse and heterogeneous human preferences. Hence, such oversimplification
limits LLMs from supporting personalization and pluralistic alignment.
Theoretically, we show that when human preferences follow a mixture
distribution of diverse subgroups, a single BT model has an irreducible error.
While existing solutions, such as multi-objective learning with fine-grained
annotations, help address this issue, they are costly and constrained by
predefined attributes, failing to fully capture the richness of human values.
In this work, we introduce MiCRo, a two-stage framework that enhances
personalized preference learning by leveraging large-scale binary preference
datasets without requiring explicit fine-grained annotations. In the first
stage, MiCRo introduces context-aware mixture modeling approach to capture
diverse human preferences. In the second stage, MiCRo integrates an online
routing strategy that dynamically adapts mixture weights based on specific
context to resolve ambiguity, allowing for efficient and scalable preference
adaptation with minimal additional supervision. Experiments on multiple
preference datasets demonstrate that MiCRo effectively captures diverse human
preferences and significantly improves downstream personalization.
MiCRo, a two-stage framework, improves personalized preference learning for large language models by leveraging binary preference datasets and dynamically adapting mixture weights based on context, effectively capturing diverse human preferences.
Reward modeling reinforcement learning from human feedback (RLHF) Large Language Models (LLMs) Bradley-Terry (BT) model mixture distribution personalization pluralistic alignment multi-objective learning context-aware mixture modeling online routing strategy