MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge
As the era of large language models (LLMs) on behalf of users unfolds,
Preference Optimization (PO) methods have become a central approach to aligning
LLMs with human preferences and improving performance. We propose Maximum a
Posteriori Preference Optimization (MaPPO), a framework for learning from
preferences that explicitly incorporates prior reward knowledge into the
optimization objective. While existing methods such as Direct Preference
Optimization (DPO) and its variants treat preference learning as a Maximum
Likelihood Estimation (MLE) problem, MaPPO extends this paradigm by integrating
prior reward estimates into a principled Maximum a Posteriori (MaP) objective.
This not only generalizes DPO and its variants, but also enhances alignment by
mitigating the oversimplified binary classification of responses. More
importantly, MaPPO introduces no additional hyperparameter, and supports
preference optimization in both offline and online settings. In addition, MaPPO
can be used as a plugin with consistent improvement on DPO variants, including
widely used SimPO, IPO, and CPO. Extensive empirical evaluations of different
model sizes and model series on three standard benchmarks, including MT-Bench,
AlpacaEval 2.0, and Arena-Hard, demonstrate consistent improvements in
alignment performance without sacrificing computational efficiency.
68897f3631e1218a08928196
Guangchen Lan
claimed_verified
2025-07-30T09:04:39.297Z
64ff4b1a0e8369f6a8c47c7e
https://cdn-avatars.huggingface.co/v1/production/uploads/64ff4b1a0e8369f6a8c47c7e/X8ocOagj89gexSa253q_8.png
Eric Lan
user
Eric-Lan
68897f3631e1218a08928197
Sipeng Zhang
68897f3631e1218a08928198
Tianle Wang
claimed_verified
2025-07-31T08:16:03.188Z
6671312e6d34fe6262a098db
/avatars/368cddf22e017e12c14225c0106c10a5.svg
Tianle Wang
user
wtl666wtl
68897f3631e1218a08928199
Yuwei Zhang
claimed_verified
2025-07-30T09:04:34.994Z
663187c1a2354b0f50ab10a0
/avatars/bef0fd9d2afa6a4990bc32bd55cbe163.svg
Yuwei Zhang
user
YWZBrandon
68897f3631e1218a0892819a
Daoan Zhang
68897f3631e1218a0892819b
Xinpeng Wei
68897f3631e1218a0892819c
Xiaoman Pan
68897f3631e1218a0892819d
Hongming Zhang
68897f3631e1218a0892819e
Dong-Jun Han
68897f3631e1218a0892819f
Christopher G. Brinton
MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge
As the era of large language models (LLMs) on behalf of users unfolds,
Preference Optimization (PO) methods have become a central approach to aligning
LLMs with human preferences and improving performance. We propose Maximum a
Posteriori Preference Optimization (MaPPO), a framework for learning from
preferences that explicitly incorporates prior reward knowledge into the
optimization objective. While existing methods such as Direct Preference
Optimization (DPO) and its variants treat preference learning as a Maximum
Likelihood Estimation (MLE) problem, MaPPO extends this paradigm by integrating
prior reward estimates into a principled Maximum a Posteriori (MaP) objective.
This not only generalizes DPO and its variants, but also enhances alignment by
mitigating the oversimplified binary classification of responses. More
importantly, MaPPO introduces no additional hyperparameter, and supports
preference optimization in both offline and online settings. In addition, MaPPO
can be used as a plugin with consistent improvement on DPO variants, including
widely used SimPO, IPO, and CPO. Extensive empirical evaluations of different
model sizes and model series on three standard benchmarks, including MT-Bench,
AlpacaEval 2.0, and Arena-Hard, demonstrate consistent improvements in
alignment performance without sacrificing computational efficiency.
MaPPO, a framework for preference optimization, enhances alignment of large language models with human preferences by integrating prior reward knowledge into a Maximum a Posteriori objective, improving performance across various benchmarks.
Maximum a Posteriori Preference Optimization MaPPO Preference Optimization Direct Preference Optimization DPO Maximum Likelihood Estimation MLE Maximum a Posteriori MaP SimPO IPO CPO MT-Bench AlpacaEval 2.0 Arena-Hard