Geometric-Mean Policy Optimization
Recent advancements, such as Group Relative Policy Optimization (GRPO), have
enhanced the reasoning capabilities of large language models by optimizing the
arithmetic mean of token-level rewards. However, GRPO suffers from unstable
policy updates when processing tokens with outlier importance-weighted rewards,
which manifests as extreme importance sampling ratios during training, i.e.,
the ratio between the sampling probabilities assigned to a token by the current
and old policies. In this work, we propose Geometric-Mean Policy Optimization
(GMPO), a stabilized variant of GRPO. Instead of optimizing the arithmetic
mean, GMPO maximizes the geometric mean of token-level rewards, which is
inherently less sensitive to outliers and maintains a more stable range of
importance sampling ratio. In addition, we provide comprehensive theoretical
and experimental analysis to justify the design and stability benefits of GMPO.
Beyond improved stability, GMPO-7B outperforms GRPO by an average of 4.1% on
multiple mathematical benchmarks and 1.4% on multimodal reasoning benchmark,
including AIME24, AMC, MATH500, OlympiadBench, Minerva, and Geometry3K. Code is
available at https://github.com/callsys/GMPO.
68883a8eaf872d625c10c631
Yuzhong Zhao
68883a8eaf872d625c10c632
Yue Liu
68883a8eaf872d625c10c633
Junpeng Liu
claimed_verified
2025-07-29T07:08:16.611Z
6512a2e284bedffb5ac5a511
/avatars/e798f2c8633d77aea41fce684d749390.svg
Junpeng Liu
user
jeepliu
68883a8eaf872d625c10c634
Jingye Chen
68883a8eaf872d625c10c635
Xun Wu
68883a8eaf872d625c10c636
Yaru Hao
68883a8eaf872d625c10c637
Tengchao Lv
68883a8eaf872d625c10c638
Shaohan Huang
68883a8eaf872d625c10c639
Lei Cui
68883a8eaf872d625c10c63a
Qixiang Ye
68883a8eaf872d625c10c63b
Fang Wan
68883a8eaf872d625c10c63c
Furu Wei
Geometric-Mean Policy Optimization
Recent advancements, such as Group Relative Policy Optimization (GRPO), have
enhanced the reasoning capabilities of large language models by optimizing the
arithmetic mean of token-level rewards. However, GRPO suffers from unstable
policy updates when processing tokens with outlier importance-weighted rewards,
which manifests as extreme importance sampling ratios during training, i.e.,
the ratio between the sampling probabilities assigned to a token by the current
and old policies. In this work, we propose Geometric-Mean Policy Optimization
(GMPO), a stabilized variant of GRPO. Instead of optimizing the arithmetic
mean, GMPO maximizes the geometric mean of token-level rewards, which is
inherently less sensitive to outliers and maintains a more stable range of
importance sampling ratio. In addition, we provide comprehensive theoretical
and experimental analysis to justify the design and stability benefits of GMPO.
Beyond improved stability, GMPO-7B outperforms GRPO by an average of 4.1% on
multiple mathematical benchmarks and 1.4% on multimodal reasoning benchmark,
including AIME24, AMC, MATH500, OlympiadBench, Minerva, and Geometry3K. Code is
available at https://github.com/callsys/GMPO.
Geometric-Mean Policy Optimization (GMPO) stabilizes policy updates in large language models by maximizing the geometric mean of token-level rewards, outperforming Group Relative Policy Optimization (GRPO) on mathematical and multimodal reasoning benchmarks.
Group Relative Policy Optimization Geometric-Mean Policy Optimization token-level rewards importance-weighted rewards importance sampling ratios GMPO GRPO AIME24 AMC MATH500 OlympiadBench Minerva Geometry3K