Perception-Aware Policy Optimization for Multimodal Reasoning
Reinforcement Learning with Verifiable Rewards (RLVR) has proven to be a
highly effective strategy for endowing Large Language Models (LLMs) with robust
multi-step reasoning abilities. However, its design and optimizations remain
tailored to purely textual domains, resulting in suboptimal performance when
applied to multimodal reasoning tasks. In particular, we observe that a major
source of error in current multimodal reasoning lies in the perception of
visual inputs. To address this bottleneck, we propose Perception-Aware Policy
Optimization (PAPO), a simple yet effective extension of GRPO that encourages
the model to learn to perceive while learning to reason, entirely from internal
supervision signals. Notably, PAPO does not rely on additional data curation,
external reward models, or proprietary models. Specifically, we introduce the
Implicit Perception Loss in the form of a KL divergence term to the GRPO
objective, which, despite its simplicity, yields significant overall
improvements (4.4%) on diverse multimodal benchmarks. The improvements are more
pronounced, approaching 8.0%, on tasks with high vision dependency. We also
observe a substantial reduction (30.5%) in perception errors, indicating
improved perceptual capabilities with PAPO. We conduct comprehensive analysis
of PAPO and identify a unique loss hacking issue, which we rigorously analyze
and mitigate through a Double Entropy Loss. Overall, our work introduces a
deeper integration of perception-aware supervision into RLVR learning
objectives and lays the groundwork for a new RL framework that encourages
visually grounded reasoning. Project page: https://mikewangwzhl.github.io/PAPO.
686f326dd938c25d68441b6a
Zhenhailong Wang
686f326dd938c25d68441b6b
Xuehang Guo
686f326dd938c25d68441b6c
Sofia Stoica
686f326dd938c25d68441b6d
Haiyang Xu
claimed_verified
2025-07-10T07:09:17.834Z
645b10e80c73ea27d13f7aca
/avatars/95e565306472a15067440b5b43e07a6f.svg
xuhaiyang
user
xhyandwyy
686f326dd938c25d68441b6e
Hongru Wang
686f326dd938c25d68441b6f
Hyeonjeong Ha
686f326dd938c25d68441b70
Xiusi Chen
686f326dd938c25d68441b71
Yangyi Chen
686f326dd938c25d68441b72
Ming Yan
686f326dd938c25d68441b73
Fei Huang
686f326dd938c25d68441b74
Heng Ji
Perception-Aware Policy Optimization for Multimodal Reasoning
Reinforcement Learning with Verifiable Rewards (RLVR) has proven to be a
highly effective strategy for endowing Large Language Models (LLMs) with robust
multi-step reasoning abilities. However, its design and optimizations remain
tailored to purely textual domains, resulting in suboptimal performance when
applied to multimodal reasoning tasks. In particular, we observe that a major
source of error in current multimodal reasoning lies in the perception of
visual inputs. To address this bottleneck, we propose Perception-Aware Policy
Optimization (PAPO), a simple yet effective extension of GRPO that encourages
the model to learn to perceive while learning to reason, entirely from internal
supervision signals. Notably, PAPO does not rely on additional data curation,
external reward models, or proprietary models. Specifically, we introduce the
Implicit Perception Loss in the form of a KL divergence term to the GRPO
objective, which, despite its simplicity, yields significant overall
improvements (4.4%) on diverse multimodal benchmarks. The improvements are more
pronounced, approaching 8.0%, on tasks with high vision dependency. We also
observe a substantial reduction (30.5%) in perception errors, indicating
improved perceptual capabilities with PAPO. We conduct comprehensive analysis
of PAPO and identify a unique loss hacking issue, which we rigorously analyze
and mitigate through a Double Entropy Loss. Overall, our work introduces a
deeper integration of perception-aware supervision into RLVR learning
objectives and lays the groundwork for a new RL framework that encourages
visually grounded reasoning. Project page: https://mikewangwzhl.github.io/PAPO.
PAPO, an extension of GRPO, enhances multimodal reasoning by integrating perception-aware supervision, leading to improved performance and reduced perception errors in tasks with high vision dependency.
Reinforcement Learning with Verifiable Rewards Large Language Models multimodal reasoning Perception-Aware Policy Optimization GRPO Implicit Perception Loss KL divergence Double Entropy Loss visually grounded reasoning